Nov 03 19:36:17.983510 ip-10-0-143-206 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 19:36:18.403959 ip-10-0-143-206 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:36:18.403959 ip-10-0-143-206 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 19:36:18.403959 ip-10-0-143-206 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:36:18.403959 ip-10-0-143-206 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:36:18.407590 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407509    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:36:18.407590 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407590    2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:36:18.407590 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407594    2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:36:18.407681 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407598    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:36:18.407681 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407601    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:36:18.407681 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407604    2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:36:18.407681 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407607    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:36:18.407681 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407612    2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:36:18.407681 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407617    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:36:18.407681 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407621    2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:36:18.407681 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407624    2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:36:18.407681 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407627    2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:36:18.407681 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407630    2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:36:18.407681 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407633    2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:36:18.407681 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407636    2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:36:18.407681 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407644    2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:36:18.407681 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407647    2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:36:18.407681 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407650    2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:36:18.407681 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407653    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:36:18.407681 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407655    2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:36:18.407681 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407658    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:36:18.407681 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407660    2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:36:18.408126 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407663    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:36:18.408126 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407666    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:36:18.408126 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407668    2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:36:18.408126 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407671    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:36:18.408126 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407674    2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:36:18.408126 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407676    2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:36:18.408126 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407679    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:36:18.408126 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407681    2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:36:18.408126 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407684    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:36:18.408126 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407687    2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:36:18.408126 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407689    2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:36:18.408126 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407692    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:36:18.408126 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407695    2566 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:36:18.408126 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407697    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:36:18.408126 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407700    2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:36:18.408126 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407702    2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:36:18.408126 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407705    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:36:18.408126 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407707    2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:36:18.408126 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407709    2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:36:18.408126 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407712    2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:36:18.408613 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407714    2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:36:18.408613 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407717    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:36:18.408613 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407719    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:36:18.408613 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407722    2566 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:36:18.408613 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407724    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:36:18.408613 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407728    2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:36:18.408613 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407731    2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:36:18.408613 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407733    2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:36:18.408613 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407735    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:36:18.408613 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407738    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:36:18.408613 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407741    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:36:18.408613 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407744    2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:36:18.408613 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407746    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:36:18.408613 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407749    2566 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:36:18.408613 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407752    2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:36:18.408613 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407754    2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:36:18.408613 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407757    2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:36:18.408613 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407759    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:36:18.408613 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407761    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:36:18.409051 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407764    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:36:18.409051 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407766    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:36:18.409051 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407768    2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:36:18.409051 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407771    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:36:18.409051 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407773    2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:36:18.409051 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407777    2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:36:18.409051 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407779    2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:36:18.409051 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407781    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:36:18.409051 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407784    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:36:18.409051 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407786    2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:36:18.409051 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407789    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:36:18.409051 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407791    2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:36:18.409051 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407794    2566 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:36:18.409051 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407797    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:36:18.409051 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407799    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:36:18.409051 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407802    2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:36:18.409051 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407804    2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:36:18.409051 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407807    2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:36:18.409051 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407810    2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:36:18.409051 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407814    2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:36:18.409542 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407818    2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:36:18.409542 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407821    2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:36:18.409542 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407824    2566 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:36:18.409542 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407827    2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:36:18.409542 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407830    2566 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:36:18.409542 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407832    2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:36:18.409542 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407835    2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:36:18.409542 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407838    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:36:18.409542 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407840    2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:36:18.409542 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407843    2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:36:18.409542 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407845    2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:36:18.409542 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407848    2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:36:18.409542 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407850    2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:36:18.409542 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407852    2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:36:18.409542 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407855    2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:36:18.409542 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407857    2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:36:18.409542 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407860    2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:36:18.409542 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407862    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:36:18.409542 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407865    2566 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:36:18.409542 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407867    2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:36:18.410055 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407869    2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:36:18.410055 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407873    2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:36:18.410055 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407876    2566 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:36:18.410055 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407878    2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:36:18.410055 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.407881    2566 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:36:18.410055 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408670    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:36:18.410055 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408677    2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:36:18.410055 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408680    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:36:18.410055 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408683    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:36:18.410055 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408685    2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:36:18.410055 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408689    2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:36:18.410055 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408693    2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:36:18.410055 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408696    2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:36:18.410055 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408698    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:36:18.410055 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408701    2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:36:18.410055 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408704    2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:36:18.410055 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408706    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:36:18.410055 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408709    2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:36:18.410055 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408712    2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:36:18.410055 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408715    2566 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:36:18.410673 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408718    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:36:18.410673 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408721    2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:36:18.410673 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408723    2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:36:18.410673 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408726    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:36:18.410673 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408728    2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:36:18.410673 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408731    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:36:18.410673 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408734    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:36:18.410673 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408737    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:36:18.410673 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408739    2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:36:18.410673 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408742    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:36:18.410673 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408744    2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:36:18.410673 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408747    2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:36:18.410673 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408749    2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:36:18.410673 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408752    2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:36:18.410673 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408754    2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:36:18.410673 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408757    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:36:18.410673 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408759    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:36:18.410673 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408762    2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:36:18.410673 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408764    2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:36:18.411146 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408767    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:36:18.411146 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408769    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:36:18.411146 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408771    2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:36:18.411146 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408774    2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:36:18.411146 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408778    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:36:18.411146 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408781    2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:36:18.411146 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408784    2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:36:18.411146 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408787    2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:36:18.411146 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408789    2566 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:36:18.411146 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408792    2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:36:18.411146 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408795    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:36:18.411146 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408798    2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:36:18.411146 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408800    2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:36:18.411146 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408803    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:36:18.411146 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408806    2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:36:18.411146 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408808    2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:36:18.411146 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408810    2566 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:36:18.411146 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408813    2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:36:18.411146 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408815    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:36:18.411146 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408818    2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:36:18.411630 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408820    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:36:18.411630 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408822    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:36:18.411630 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408825    2566 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:36:18.411630 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408827    2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:36:18.411630 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408830    2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:36:18.411630 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408832    2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:36:18.411630 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408835    2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:36:18.411630 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408839    2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:36:18.411630 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408843    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:36:18.411630 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408846    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:36:18.411630 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408848    2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:36:18.411630 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408851    2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:36:18.411630 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408854    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:36:18.411630 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408856    2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:36:18.411630 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408859    2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:36:18.411630 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408862    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:36:18.411630 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408864    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:36:18.411630 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408867    2566 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:36:18.411630 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408870    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:36:18.412085 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408872    2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:36:18.412085 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408875    2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:36:18.412085 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408877    2566 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:36:18.412085 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408880    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:36:18.412085 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408883    2566 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:36:18.412085 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408885    2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:36:18.412085 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408888    2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:36:18.412085 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408890    2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:36:18.412085 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408893    2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:36:18.412085 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408895    2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:36:18.412085 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408898    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:36:18.412085 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408900    2566 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:36:18.412085 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408903    2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:36:18.412085 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408905    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:36:18.412085 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408907    2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:36:18.412085 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408911    2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:36:18.412085 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408915    2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:36:18.412085 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408918    2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:36:18.412085 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408921    2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:36:18.412085 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408924    2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:36:18.412584 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408926    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:36:18.412584 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408928    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:36:18.412584 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408931    2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:36:18.412584 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408933    2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:36:18.412584 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408936    2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:36:18.412584 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408938    2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:36:18.412584 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408941    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:36:18.412584 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408945    2566 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:36:18.412584 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408948    2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:36:18.412584 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408950    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:36:18.412584 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408953    2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:36:18.412584 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408955    2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:36:18.412584 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.408958    2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:36:18.412584 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409454    2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:36:18.412584 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409460    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:36:18.412584 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409463    2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:36:18.412584 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409466    2566 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:36:18.412584 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409468    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:36:18.412584 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409471    2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:36:18.412584 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409473    2566 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:36:18.413052 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409476    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:36:18.413052 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409478    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:36:18.413052 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409481    2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:36:18.413052 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409483    2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:36:18.413052 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409486    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:36:18.413052 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409488    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:36:18.413052 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409491    2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:36:18.413052 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409493    2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:36:18.413052 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409496    2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:36:18.413052 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409502    2566 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:36:18.413052 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409505    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:36:18.413052 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409507    2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:36:18.413052 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409510    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:36:18.413052 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409512    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:36:18.413052 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409516    2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:36:18.413052 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409520    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:36:18.413052 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409522    2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:36:18.413052 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409525    2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:36:18.413052 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409528    2566 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:36:18.413517 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409531    2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:36:18.413517 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409533    2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:36:18.413517 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409537    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:36:18.413517 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409539    2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:36:18.413517 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409542    2566 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:36:18.413517 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409545    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:36:18.413517 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409548    2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:36:18.413517 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409551    2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:36:18.413517 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409553    2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:36:18.413517 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409557    2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:36:18.413517 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409561    2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:36:18.413517 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409564    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:36:18.413517 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409567    2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:36:18.413517 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409570    2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:36:18.413517 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409572    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:36:18.413517 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409574    2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:36:18.413517 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409577    2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:36:18.413517 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409579    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:36:18.413517 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409582    2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:36:18.414016 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409585    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:36:18.414016 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409588    2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:36:18.414016 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409590    2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:36:18.414016 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409593    2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:36:18.414016 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409596    2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:36:18.414016 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409599    2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:36:18.414016 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409601    2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:36:18.414016 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409604    2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:36:18.414016 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409606    2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:36:18.414016 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409609    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:36:18.414016 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409611    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:36:18.414016 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409614    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:36:18.414016 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409616    2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:36:18.414016 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409618    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:36:18.414016 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409621    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:36:18.414016 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409624    2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:36:18.414016 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409627    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:36:18.414016 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409629    2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:36:18.414016 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409632    2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:36:18.414496 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409635    2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:36:18.414496 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409637    2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:36:18.414496 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409640    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:36:18.414496 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409642    2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:36:18.414496 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409645    2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:36:18.414496 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409647    2566 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:36:18.414496 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409649    2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:36:18.414496 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409652    2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:36:18.414496 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409654    2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:36:18.414496 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409657    2566 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:36:18.414496 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409659    2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:36:18.414496 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409661    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:36:18.414496 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409664    2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:36:18.414496 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409666    2566 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:36:18.414496 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409668    2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:36:18.414496 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409671    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:36:18.414496 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409674    2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:36:18.414496 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409676    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:36:18.414496 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409679    2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:36:18.414496 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409682    2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:36:18.414951 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409685    2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:36:18.414951 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409687    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:36:18.414951 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409690    2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:36:18.414951 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409692    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:36:18.414951 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409695    2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:36:18.414951 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409697    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:36:18.414951 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409700    2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:36:18.414951 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409702    2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:36:18.414951 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409704    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:36:18.414951 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409707    2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:36:18.414951 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409709    2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:36:18.414951 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409712    2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:36:18.414951 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409714    2566 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:36:18.414951 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409717    2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:36:18.414951 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409720    2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:36:18.414951 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409724    2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:36:18.414951 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409726    2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:36:18.414951 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409729    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:36:18.414951 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409731    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:36:18.414951 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409734    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:36:18.415488 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409736    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:36:18.415488 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.409738    2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:36:18.415488 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411434    2566 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 19:36:18.415488 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411444    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 19:36:18.415488 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411450    2566 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 19:36:18.415488 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411455    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 19:36:18.415488 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411460    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 19:36:18.415488 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411464    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 19:36:18.415488 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411468    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 19:36:18.415488 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411472    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 19:36:18.415488 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411475    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 19:36:18.415488 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411478    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 19:36:18.415488 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411482    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 19:36:18.415488 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411485    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 19:36:18.415488 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411488    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 19:36:18.415488 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411491    2566 flags.go:64] FLAG: --cgroup-root=""
Nov 03 19:36:18.415488 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411494    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 19:36:18.415488 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411497    2566 flags.go:64] FLAG: --client-ca-file=""
Nov 03 19:36:18.415488 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411499    2566 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 19:36:18.415488 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411502    2566 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 19:36:18.415488 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411516    2566 flags.go:64] FLAG: --cluster-domain=""
Nov 03 19:36:18.415488 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411519    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 19:36:18.415488 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411522    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 19:36:18.415488 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411526    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 19:36:18.416037 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411529    2566 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 19:36:18.416037 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411538    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 19:36:18.416037 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411541    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 19:36:18.416037 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411545    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 19:36:18.416037 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411548    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 19:36:18.416037 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411551    2566 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 19:36:18.416037 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411554    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 19:36:18.416037 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411557    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 19:36:18.416037 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411560    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 19:36:18.416037 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411563    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 19:36:18.416037 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411567    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 19:36:18.416037 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411570    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 19:36:18.416037 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411573    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 19:36:18.416037 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411575    2566 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 19:36:18.416037 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411578    2566 flags.go:64] FLAG: --enable-server="true"
Nov 03 19:36:18.416037 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411582    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 19:36:18.416037 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411586    2566 flags.go:64] FLAG: --event-burst="100"
Nov 03 19:36:18.416037 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411589    2566 flags.go:64] FLAG: --event-qps="50"
Nov 03 19:36:18.416037 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411592    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 19:36:18.416037 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411595    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 19:36:18.416037 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411598    2566 flags.go:64] FLAG: --eviction-hard=""
Nov 03 19:36:18.416037 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411603    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 19:36:18.416037 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411605    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 19:36:18.416037 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411608    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 19:36:18.416037 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411611    2566 flags.go:64] FLAG: --eviction-soft=""
Nov 03 19:36:18.416726 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411614    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 19:36:18.416726 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411617    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 19:36:18.416726 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411620    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 19:36:18.416726 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411622    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 19:36:18.416726 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411625    2566 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 19:36:18.416726 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411628    2566 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 19:36:18.416726 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411631    2566 flags.go:64] FLAG: --feature-gates=""
Nov 03 19:36:18.416726 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411635    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 19:36:18.416726 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411638    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 19:36:18.416726 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411643    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 19:36:18.416726 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411646    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 19:36:18.416726 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411649    2566 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 19:36:18.416726 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411652    2566 flags.go:64] FLAG: --help="false"
Nov 03 19:36:18.416726 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411655    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:36:18.416726 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411658    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 19:36:18.416726 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411661    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 19:36:18.416726 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411664    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 19:36:18.416726 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411667    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 19:36:18.416726 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411671    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 19:36:18.416726 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411673    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 19:36:18.416726 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411678    2566 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 19:36:18.416726 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411681    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 19:36:18.416726 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411684    2566 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 19:36:18.417290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411686    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 19:36:18.417290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411690    2566 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 19:36:18.417290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411692    2566 flags.go:64] FLAG: --kube-reserved=""
Nov 03 19:36:18.417290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411696    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 19:36:18.417290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411698    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 19:36:18.417290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411701    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 19:36:18.417290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411705    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 19:36:18.417290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411707    2566 flags.go:64] FLAG: --lock-file=""
Nov 03 19:36:18.417290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411710    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 19:36:18.417290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411713    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 19:36:18.417290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411716    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 19:36:18.417290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411722    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 19:36:18.417290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411725    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 19:36:18.417290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411728    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 19:36:18.417290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411731    2566 flags.go:64] FLAG: --logging-format="text"
Nov 03 19:36:18.417290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411734    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 19:36:18.417290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411737    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 19:36:18.417290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411740    2566 flags.go:64] FLAG: --manifest-url=""
Nov 03 19:36:18.417290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411743    2566 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 19:36:18.417290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411749    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 19:36:18.417290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411752    2566 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 19:36:18.417290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411755    2566 flags.go:64] FLAG: --max-pods="110"
Nov 03 19:36:18.417290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411758    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 19:36:18.417290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411761    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 19:36:18.417290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411764    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 19:36:18.417866 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411767    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 19:36:18.417866 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411770    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 19:36:18.417866 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411772    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 19:36:18.417866 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411775    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 19:36:18.417866 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411784    2566 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 19:36:18.417866 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411788    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 19:36:18.417866 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411791    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 19:36:18.417866 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411794    2566 flags.go:64] FLAG: --pod-cidr=""
Nov 03 19:36:18.417866 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411796    2566 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 19:36:18.417866 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411799    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 19:36:18.417866 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411802    2566 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 19:36:18.417866 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411805    2566 flags.go:64] FLAG: --port="10250"
Nov 03 19:36:18.417866 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411808    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 19:36:18.417866 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411811    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0b788c8e870c8382f"
Nov 03 19:36:18.417866 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411814    2566 flags.go:64] FLAG: --qos-reserved=""
Nov 03 19:36:18.417866 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411817    2566 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 19:36:18.417866 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411820    2566 flags.go:64] FLAG: --register-node="true"
Nov 03 19:36:18.417866 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411823    2566 flags.go:64] FLAG: --register-with-taints=""
Nov 03 19:36:18.417866 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411827    2566 flags.go:64] FLAG: --registry-burst="10"
Nov 03 19:36:18.417866 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411830    2566 flags.go:64] FLAG: --registry-qps="5"
Nov 03 19:36:18.417866 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411833    2566 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 19:36:18.417866 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411835    2566 flags.go:64] FLAG: --reserved-memory=""
Nov 03 19:36:18.417866 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411839    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 19:36:18.417866 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411842    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 19:36:18.417866 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411846    2566 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 19:36:18.417866 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411849    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 19:36:18.418535 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411852    2566 flags.go:64] FLAG: --runonce="false"
Nov 03 19:36:18.418535 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411855    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 19:36:18.418535 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411858    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 19:36:18.418535 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411861    2566 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 19:36:18.418535 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411864    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 19:36:18.418535 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411867    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 19:36:18.418535 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411869    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 19:36:18.418535 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411872    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 19:36:18.418535 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411875    2566 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 19:36:18.418535 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411878    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 19:36:18.418535 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411882    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 19:36:18.418535 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411885    2566 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 19:36:18.418535 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411888    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 19:36:18.418535 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411891    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 19:36:18.418535 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411894    2566 flags.go:64] FLAG: --system-cgroups=""
Nov 03 19:36:18.418535 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411897    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 19:36:18.418535 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411902    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 19:36:18.418535 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411905    2566 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 19:36:18.418535 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411908    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 19:36:18.418535 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411912    2566 flags.go:64] FLAG: --tls-min-version=""
Nov 03 19:36:18.418535 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411915    2566 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 19:36:18.418535 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411918    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 19:36:18.418535 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411921    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 19:36:18.418535 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411924    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 19:36:18.418535 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411926    2566 flags.go:64] FLAG: --v="2"
Nov 03 19:36:18.419123 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411931    2566 flags.go:64] FLAG: --version="false"
Nov 03 19:36:18.419123 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411935    2566 flags.go:64] FLAG: --vmodule=""
Nov 03 19:36:18.419123 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411939    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 19:36:18.419123 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.411942    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 19:36:18.419123 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412104    2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:36:18.419123 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412107    2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:36:18.419123 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412111    2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:36:18.419123 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412114    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:36:18.419123 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412117    2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:36:18.419123 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412125    2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:36:18.419123 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412128    2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:36:18.419123 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412144    2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:36:18.419123 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412148    2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:36:18.419123 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412150    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:36:18.419123 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412153    2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:36:18.419123 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412156    2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:36:18.419123 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412158    2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:36:18.419123 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412161    2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:36:18.419123 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412164    2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:36:18.419123 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412167    2566 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:36:18.419620 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412169    2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:36:18.419620 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412171    2566 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:36:18.419620 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412174    2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:36:18.419620 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412177    2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:36:18.419620 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412180    2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:36:18.419620 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412183    2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:36:18.419620 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412185    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:36:18.419620 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412188    2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:36:18.419620 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412190    2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:36:18.419620 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412193    2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:36:18.419620 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412195    2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:36:18.419620 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412199    2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:36:18.419620 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412203    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:36:18.419620 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412207    2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:36:18.419620 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412210    2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:36:18.419620 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412213    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:36:18.419620 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412215    2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:36:18.419620 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412218    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:36:18.419620 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412221    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:36:18.420064 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412224    2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:36:18.420064 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412226    2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:36:18.420064 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412229    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:36:18.420064 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412238    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:36:18.420064 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412241    2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:36:18.420064 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412243    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:36:18.420064 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412246    2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:36:18.420064 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412248    2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:36:18.420064 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412251    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:36:18.420064 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412253    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:36:18.420064 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412256    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:36:18.420064 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412258    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:36:18.420064 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412261    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:36:18.420064 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412263    2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:36:18.420064 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412266    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:36:18.420064 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412269    2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:36:18.420064 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412271    2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:36:18.420064 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412274    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:36:18.420064 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412276    2566 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:36:18.420525 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412279    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:36:18.420525 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412281    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:36:18.420525 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412284    2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:36:18.420525 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412286    2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:36:18.420525 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412288    2566 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:36:18.420525 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412291    2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:36:18.420525 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412293    2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:36:18.420525 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412296    2566 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:36:18.420525 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412298    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:36:18.420525 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412301    2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:36:18.420525 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412303    2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:36:18.420525 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412306    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:36:18.420525 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412309    2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:36:18.420525 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412312    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:36:18.420525 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412314    2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:36:18.420525 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412317    2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:36:18.420525 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412319    2566 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:36:18.420525 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412329    2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:36:18.420525 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412331    2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:36:18.420525 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412334    2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:36:18.420999 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412336    2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:36:18.420999 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412339    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:36:18.420999 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412341    2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:36:18.420999 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412343    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:36:18.420999 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412346    2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:36:18.420999 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412348    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:36:18.420999 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412351    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:36:18.420999 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412353    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:36:18.420999 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412355    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:36:18.420999 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412358    2566 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:36:18.420999 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412360    2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:36:18.420999 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412363    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:36:18.420999 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412365    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:36:18.420999 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412367    2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:36:18.420999 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412370    2566 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:36:18.420999 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412372    2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:36:18.420999 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412374    2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:36:18.420999 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412377    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:36:18.420999 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412385    2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:36:18.420999 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412388    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:36:18.421478 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412390    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:36:18.421478 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412393    2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:36:18.421478 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412395    2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:36:18.421478 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412398    2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:36:18.421478 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412402    2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:36:18.421478 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412405    2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:36:18.421478 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412407    2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:36:18.421478 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412410    2566 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:36:18.421478 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412413    2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:36:18.421478 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412417    2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:36:18.421478 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412425    2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:36:18.421478 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412428    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:36:18.421478 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.412457    2566 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:36:18.421478 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412509    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:36:18.421478 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412513    2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:36:18.421829 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412516    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:36:18.421829 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412519    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:36:18.421829 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412522    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:36:18.421829 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412524    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:36:18.421829 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412527    2566 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:36:18.421829 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412529    2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:36:18.421829 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412532    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:36:18.421829 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412534    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:36:18.421829 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412536    2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:36:18.421829 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412539    2566 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:36:18.421829 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412541    2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:36:18.421829 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412544    2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:36:18.421829 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412546    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:36:18.421829 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412548    2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:36:18.421829 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412551    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:36:18.421829 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412553    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:36:18.421829 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412555    2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:36:18.421829 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412559    2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:36:18.421829 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412562    2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:36:18.421829 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412564    2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:36:18.422304 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412567    2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:36:18.422304 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412571    2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:36:18.422304 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412574    2566 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:36:18.422304 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412577    2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:36:18.422304 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412581    2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:36:18.422304 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412583    2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:36:18.422304 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412586    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:36:18.422304 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412589    2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:36:18.422304 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412597    2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:36:18.422304 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412600    2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:36:18.422304 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412602    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:36:18.422304 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412604    2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:36:18.422304 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412607    2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:36:18.422304 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412610    2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:36:18.422304 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412612    2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:36:18.422304 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412615    2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:36:18.422304 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412617    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:36:18.422304 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412620    2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:36:18.422304 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412622    2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:36:18.422753 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412625    2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:36:18.422753 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412627    2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:36:18.422753 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412629    2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:36:18.422753 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412632    2566 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:36:18.422753 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412634    2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:36:18.422753 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412637    2566 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:36:18.422753 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412639    2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:36:18.422753 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412641    2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:36:18.422753 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412645    2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:36:18.422753 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412647    2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:36:18.422753 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412651    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:36:18.422753 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412654    2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:36:18.422753 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412659    2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:36:18.422753 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412662    2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:36:18.422753 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412666    2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:36:18.422753 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412671    2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:36:18.422753 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412675    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:36:18.422753 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412678    2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:36:18.422753 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412681    2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:36:18.422753 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412683    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:36:18.423237 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412686    2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:36:18.423237 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412689    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:36:18.423237 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412698    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:36:18.423237 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412701    2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:36:18.423237 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412703    2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:36:18.423237 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412706    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:36:18.423237 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412708    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:36:18.423237 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412711    2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:36:18.423237 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412713    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:36:18.423237 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412715    2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:36:18.423237 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412718    2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:36:18.423237 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412720    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:36:18.423237 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412723    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:36:18.423237 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412725    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:36:18.423237 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412728    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:36:18.423237 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412730    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:36:18.423237 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412733    2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:36:18.423237 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412735    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:36:18.423237 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412738    2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:36:18.423677 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412740    2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:36:18.423677 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412743    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:36:18.423677 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412745    2566 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:36:18.423677 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412749    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:36:18.423677 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412751    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:36:18.423677 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412754    2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:36:18.423677 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412756    2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:36:18.423677 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412760    2566 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:36:18.423677 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412763    2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:36:18.423677 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412766    2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:36:18.423677 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412768    2566 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:36:18.423677 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412771    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:36:18.423677 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412773    2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:36:18.423677 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412775    2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:36:18.423677 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412778    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:36:18.423677 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412780    2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:36:18.423677 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412788    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:36:18.423677 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412792    2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:36:18.423677 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412794    2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:36:18.423677 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412797    2566 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:36:18.424193 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412799    2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:36:18.424193 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412802    2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:36:18.424193 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412804    2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:36:18.424193 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412806    2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:36:18.424193 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412809    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:36:18.424193 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.412811    2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:36:18.424193 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.419977    2566 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 19:36:18.424193 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.419991    2566 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 19:36:18.424193 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.420767    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 19:36:18.424193 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.420774    2566 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 19:36:18.424193 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420861    2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:36:18.424193 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420867    2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:36:18.424193 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420870    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:36:18.424193 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420873    2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:36:18.424193 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420876    2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:36:18.424193 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420878    2566 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:36:18.424193 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420881    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:36:18.424193 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420884    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:36:18.424193 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420886    2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:36:18.424193 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420889    2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:36:18.424660 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420892    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:36:18.424660 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420894    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:36:18.424660 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420897    2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:36:18.424660 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420899    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:36:18.424660 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420902    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:36:18.424660 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420905    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:36:18.424660 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420907    2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:36:18.424660 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420910    2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:36:18.424660 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420913    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:36:18.424660 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420916    2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:36:18.424660 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420918    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:36:18.424660 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420921    2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:36:18.424660 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420923    2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:36:18.424660 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420925    2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:36:18.424660 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420928    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:36:18.424660 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420930    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:36:18.424660 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420933    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:36:18.424660 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420935    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:36:18.424660 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420939    2566 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:36:18.424660 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420942    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:36:18.425116 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420944    2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:36:18.425116 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420947    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:36:18.425116 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420949    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:36:18.425116 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420952    2566 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:36:18.425116 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420954    2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:36:18.425116 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420957    2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:36:18.425116 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420959    2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:36:18.425116 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420962    2566 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:36:18.425116 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420964    2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:36:18.425116 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420967    2566 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:36:18.425116 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420970    2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:36:18.425116 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420972    2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:36:18.425116 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420975    2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:36:18.425116 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420977    2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:36:18.425116 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420979    2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:36:18.425116 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420982    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:36:18.425116 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420985    2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:36:18.425116 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420987    2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:36:18.425116 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420990    2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:36:18.425116 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420992    2566 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:36:18.425116 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420995    2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:36:18.425616 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.420998    2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:36:18.425616 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421000    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:36:18.425616 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421003    2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:36:18.425616 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421005    2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:36:18.425616 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421008    2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:36:18.425616 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421011    2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:36:18.425616 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421014    2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:36:18.425616 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421016    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:36:18.425616 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421020    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:36:18.425616 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421023    2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:36:18.425616 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421025    2566 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:36:18.425616 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421028    2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:36:18.425616 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421030    2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:36:18.425616 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421033    2566 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:36:18.425616 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421036    2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:36:18.425616 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421038    2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:36:18.425616 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421041    2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:36:18.425616 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421043    2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:36:18.425616 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421046    2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:36:18.426057 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421049    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:36:18.426057 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421051    2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:36:18.426057 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421053    2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:36:18.426057 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421056    2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:36:18.426057 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421058    2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:36:18.426057 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421061    2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:36:18.426057 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421065    2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:36:18.426057 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421068    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:36:18.426057 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421071    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:36:18.426057 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421074    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:36:18.426057 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421077    2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:36:18.426057 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421079    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:36:18.426057 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421081    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:36:18.426057 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421084    2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:36:18.426057 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421087    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:36:18.426057 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421089    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:36:18.426057 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421092    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:36:18.426057 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421094    2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:36:18.426057 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421097    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:36:18.426510 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421099    2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:36:18.426510 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421102    2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:36:18.426510 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421105    2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:36:18.426510 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421109    2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:36:18.426510 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421113    2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:36:18.426510 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421116    2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:36:18.426510 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421118    2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:36:18.426510 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421121    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:36:18.426510 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421123    2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:36:18.426510 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421126    2566 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:36:18.426510 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421128    2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:36:18.426510 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421142    2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:36:18.426510 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421145    2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:36:18.426510 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421148    2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:36:18.426510 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421150    2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:36:18.426510 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421153    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:36:18.426510 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421155    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:36:18.426912 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.421184    2566 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:36:18.426912 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421325    2566 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:36:18.426912 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421330    2566 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:36:18.426912 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421332    2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:36:18.426912 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421335    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:36:18.426912 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421337    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:36:18.426912 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421340    2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:36:18.426912 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421342    2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:36:18.426912 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421345    2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:36:18.426912 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421349    2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:36:18.426912 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421353    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:36:18.426912 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421357    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:36:18.426912 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421360    2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:36:18.426912 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421362    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:36:18.426912 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421365    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:36:18.427285 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421368    2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:36:18.427285 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421370    2566 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:36:18.427285 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421373    2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:36:18.427285 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421375    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:36:18.427285 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421377    2566 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:36:18.427285 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421380    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:36:18.427285 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421382    2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:36:18.427285 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421385    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:36:18.427285 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421387    2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:36:18.427285 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421390    2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:36:18.427285 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421392    2566 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:36:18.427285 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421395    2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:36:18.427285 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421397    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:36:18.427285 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421399    2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:36:18.427285 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421402    2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:36:18.427285 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421404    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:36:18.427285 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421407    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:36:18.427285 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421409    2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:36:18.427285 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421411    2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:36:18.427732 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421414    2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:36:18.427732 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421417    2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:36:18.427732 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421419    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:36:18.427732 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421421    2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:36:18.427732 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421424    2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:36:18.427732 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421426    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:36:18.427732 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421429    2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:36:18.427732 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421431    2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:36:18.427732 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421433    2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:36:18.427732 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421436    2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:36:18.427732 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421439    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:36:18.427732 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421442    2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:36:18.427732 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421444    2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:36:18.427732 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421447    2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:36:18.427732 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421449    2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:36:18.427732 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421452    2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:36:18.427732 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421454    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:36:18.427732 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421456    2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:36:18.427732 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421459    2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:36:18.427732 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421461    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:36:18.428333 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421464    2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:36:18.428333 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421466    2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:36:18.428333 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421468    2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:36:18.428333 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421471    2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:36:18.428333 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421473    2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:36:18.428333 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421475    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:36:18.428333 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421478    2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:36:18.428333 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421481    2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:36:18.428333 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421483    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:36:18.428333 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421486    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:36:18.428333 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421488    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:36:18.428333 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421491    2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:36:18.428333 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421494    2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:36:18.428333 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421496    2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:36:18.428333 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421498    2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:36:18.428333 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421501    2566 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:36:18.428333 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421503    2566 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:36:18.428333 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421506    2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:36:18.428333 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421508    2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:36:18.428333 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421510    2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:36:18.428808 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421513    2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:36:18.428808 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421515    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:36:18.428808 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421517    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:36:18.428808 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421520    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:36:18.428808 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421523    2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:36:18.428808 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421525    2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:36:18.428808 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421528    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:36:18.428808 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421530    2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:36:18.428808 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421533    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:36:18.428808 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421535    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:36:18.428808 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421537    2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:36:18.428808 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421540    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:36:18.428808 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421542    2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:36:18.428808 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421546    2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:36:18.428808 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421549    2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:36:18.428808 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421552    2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:36:18.428808 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421554    2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:36:18.428808 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421557    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:36:18.428808 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421559    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:36:18.429316 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421562    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:36:18.429316 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421564    2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:36:18.429316 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421566    2566 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:36:18.429316 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421569    2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:36:18.429316 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421571    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:36:18.429316 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421574    2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:36:18.429316 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421576    2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:36:18.429316 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421579    2566 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:36:18.429316 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421581    2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:36:18.429316 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421583    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:36:18.429316 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421586    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:36:18.429316 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421588    2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:36:18.429316 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421591    2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:36:18.429316 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421593    2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:36:18.429316 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.421615    2566 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:36:18.429316 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421646    2566 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:36:18.429904 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421650    2566 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:36:18.429904 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421654    2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:36:18.429904 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421656    2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:36:18.429904 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421660    2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:36:18.429904 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421662    2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:36:18.429904 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421665    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:36:18.429904 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421667    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:36:18.429904 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421669    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:36:18.429904 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421672    2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:36:18.429904 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421674    2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:36:18.429904 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421677    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:36:18.429904 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421680    2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:36:18.429904 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421682    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:36:18.429904 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421684    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:36:18.429904 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421687    2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:36:18.429904 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421689    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:36:18.429904 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421692    2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:36:18.429904 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421695    2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:36:18.429904 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421698    2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:36:18.430486 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421701    2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:36:18.430486 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421704    2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:36:18.430486 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421706    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:36:18.430486 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421709    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:36:18.430486 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421711    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:36:18.430486 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421714    2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:36:18.430486 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421716    2566 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:36:18.430486 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421719    2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:36:18.430486 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421721    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:36:18.430486 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421724    2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:36:18.430486 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421726    2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:36:18.430486 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421729    2566 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:36:18.430486 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421731    2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:36:18.430486 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421733    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:36:18.430486 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421736    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:36:18.430486 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421739    2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:36:18.430486 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421741    2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:36:18.430486 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421744    2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:36:18.430486 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421748    2566 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:36:18.430486 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421751    2566 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:36:18.431155 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421753    2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:36:18.431155 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421755    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:36:18.431155 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421758    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:36:18.431155 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421760    2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:36:18.431155 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421763    2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:36:18.431155 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421765    2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:36:18.431155 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421770    2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:36:18.431155 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421773    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:36:18.431155 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421776    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:36:18.431155 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421779    2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:36:18.431155 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421781    2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:36:18.431155 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421783    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:36:18.431155 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421786    2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:36:18.431155 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421788    2566 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:36:18.431155 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421791    2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:36:18.431155 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421793    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:36:18.431155 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421795    2566 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:36:18.431155 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421798    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:36:18.431155 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421800    2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:36:18.431621 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421802    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:36:18.431621 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421805    2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:36:18.431621 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421807    2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:36:18.431621 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421810    2566 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:36:18.431621 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421812    2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:36:18.431621 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421815    2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:36:18.431621 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421817    2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:36:18.431621 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421820    2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:36:18.431621 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421822    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:36:18.431621 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421825    2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:36:18.431621 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421828    2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:36:18.431621 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421832    2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:36:18.431621 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421835    2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:36:18.431621 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421838    2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:36:18.431621 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421840    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:36:18.431621 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421843    2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:36:18.431621 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421845    2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:36:18.431621 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421847    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:36:18.431621 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421850    2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:36:18.431621 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421853    2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:36:18.432096 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421855    2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:36:18.432096 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421857    2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:36:18.432096 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421860    2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:36:18.432096 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421862    2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:36:18.432096 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421864    2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:36:18.432096 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421867    2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:36:18.432096 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421869    2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:36:18.432096 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421871    2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:36:18.432096 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421874    2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:36:18.432096 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421876    2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:36:18.432096 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421879    2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:36:18.432096 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421881    2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:36:18.432096 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421883    2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:36:18.432096 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421886    2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:36:18.432096 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421888    2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:36:18.432096 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421890    2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:36:18.432096 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421893    2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:36:18.432096 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421895    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:36:18.432096 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421898    2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:36:18.432096 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421901    2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:36:18.432710 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421904    2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:36:18.432710 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421906    2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:36:18.432710 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421909    2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:36:18.432710 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421912    2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:36:18.432710 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421915    2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:36:18.432710 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421917    2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:36:18.432710 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:18.421919    2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:36:18.432710 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.421972    2566 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 19:36:18.432710 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.424487    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 19:36:18.432710 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.425611    2566 server.go:1014] "Starting client certificate rotation"
Nov 03 19:36:18.432710 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.425716    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 19:36:18.432710 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.425762    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 19:36:18.432710 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.427781    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:36:18.434155 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.434116    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:36:18.446553 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.446534    2566 log.go:25] "Validated CRI v1 runtime API"
Nov 03 19:36:18.450605 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.450584    2566 log.go:25] "Validated CRI v1 image API"
Nov 03 19:36:18.451459 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.451434    2566 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 19:36:18.455042 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.455007    2566 fs.go:135] Filesystem UUIDs: map[4b595758-3b71-4bf0-b413-11dd05eb24ce:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e4defb7b-9f8b-4203-b68e-c808c060f10e:/dev/nvme0n1p3]
Nov 03 19:36:18.455122 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.455038    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 19:36:18.460351 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.460235    2566 manager.go:217] Machine: {Timestamp:2026-11-03 19:36:18.458810461 +0000 UTC m=+0.366070914 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100556 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b8201bdaa17c40acbee899fdb840b SystemUUID:ec2b8201-bdaa-17c4-0acb-ee899fdb840b BootID:f4b3fa21-120f-4523-9e14-9e963575c8f0 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:db:50:90:7f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:db:50:90:7f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:b6:a7:44:5a:86 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 19:36:18.460351 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.460348    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 19:36:18.460469 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.460435    2566 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 19:36:18.461651 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.461625    2566 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 19:36:18.461797 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.461653    2566 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-206.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 19:36:18.461848 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.461800    2566 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 19:36:18.461848 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.461808    2566 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 19:36:18.461848 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.461843    2566 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:36:18.461923 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.461861    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:36:18.463121 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.463110    2566 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 19:36:18.467680 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.467666    2566 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 19:36:18.467813 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.467804    2566 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 19:36:18.467848 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.467836    2566 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 19:36:18.467877 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.467853    2566 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 19:36:18.467877 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.467867    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 19:36:18.467877 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.467876    2566 kubelet.go:405] "Adding apiserver pod source"
Nov 03 19:36:18.467958 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.467884    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 19:36:18.468496 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.468473    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:36:18.468967 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.468937    2566 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 19:36:18.472312 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.472288    2566 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 19:36:18.480149 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.480116    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:36:18.481195 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.481181    2566 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 19:36:18.481267 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.481223    2566 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 19:36:18.481415 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.481403    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 19:36:18.481489 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.481425    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 19:36:18.481489 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.481434    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 19:36:18.481489 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.481444    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 19:36:18.481489 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.481453    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 19:36:18.481489 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.481461    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 19:36:18.482216 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.482205    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 19:36:18.482269 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.482221    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 19:36:18.482269 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.482234    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 19:36:18.482269 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.482244    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 19:36:18.482269 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.482258    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 19:36:18.482416 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.482274    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 19:36:18.483677 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.483666    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 19:36:18.483727 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.483680    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 19:36:18.486713 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:18.486664    2566 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 19:36:18.486713 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.486683    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-206.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 19:36:18.486904 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:18.486743    2566 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-206.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 19:36:18.488479 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.488465    2566 server.go:1263] "Started kubelet"
Nov 03 19:36:18.488569 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.488540    2566 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 19:36:18.488670 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.488592    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 19:36:18.488798 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.488693    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 19:36:18.489353 ip-10-0-143-206 systemd[1]: Started Kubernetes Kubelet.
Nov 03 19:36:18.490388 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.490372    2566 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 19:36:18.490697 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.490686    2566 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 19:36:18.493688 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:18.493669    2566 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 19:36:18.493847 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.493827    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wfk55"
Nov 03 19:36:18.497720 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:18.496570    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-206.us-east-2.compute.internal.18e4a018741978f5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-206.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-206.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 19:36:18.487785717 +0000 UTC m=+0.395046168,LastTimestamp:2026-11-03 19:36:18.487785717 +0000 UTC m=+0.395046168,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-206.us-east-2.compute.internal,}"
Nov 03 19:36:18.498413 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.498399    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 19:36:18.499865 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.499849    2566 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 19:36:18.500005 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:18.499964    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-206.us-east-2.compute.internal\" not found"
Nov 03 19:36:18.500724 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.500707    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 19:36:18.500724 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.500709    2566 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 19:36:18.500845 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.500734    2566 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 19:36:18.500845 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.500839    2566 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 19:36:18.500845 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.500846    2566 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 19:36:18.501590 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.501576    2566 factory.go:55] Registering systemd factory
Nov 03 19:36:18.501590 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.501592    2566 factory.go:223] Registration of the systemd container factory successfully
Nov 03 19:36:18.501812 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.501796    2566 factory.go:153] Registering CRI-O factory
Nov 03 19:36:18.501812 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.501813    2566 factory.go:223] Registration of the crio container factory successfully
Nov 03 19:36:18.501920 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.501862    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 19:36:18.501920 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.501876    2566 factory.go:103] Registering Raw factory
Nov 03 19:36:18.501920 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.501887    2566 manager.go:1199] Started watching for new ooms in manager
Nov 03 19:36:18.503726 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.503709    2566 manager.go:319] Starting recovery of all containers
Nov 03 19:36:18.505308 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.505284    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wfk55"
Nov 03 19:36:18.508988 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:18.508950    2566 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-206.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 19:36:18.516065 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.516048    2566 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:36:18.516313 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.516298    2566 manager.go:324] Recovery completed
Nov 03 19:36:18.520808 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.520795    2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:36:18.522527 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.522510    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-206.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:36:18.522591 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.522544    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-206.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:36:18.522591 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.522558    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-206.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:36:18.523093 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.523080    2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:36:18.523157 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.523080    2566 cpu_manager.go:226] "Starting" policy="none"
Nov 03 19:36:18.523157 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.523106    2566 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 19:36:18.523157 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.523124    2566 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 19:36:18.524947 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.524931    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-206.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:36:18.524997 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.524963    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-206.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:36:18.524997 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.524974    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-206.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:36:18.525123 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.525113    2566 policy_none.go:50] "Start"
Nov 03 19:36:18.525176 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.525149    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 19:36:18.525176 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.525163    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 19:36:18.527168 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.527157    2566 policy_none.go:44] "Start"
Nov 03 19:36:18.581199 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.569825    2566 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 19:36:18.581199 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.574293    2566 manager.go:341] "Starting Device Plugin manager"
Nov 03 19:36:18.581199 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:18.574321    2566 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 19:36:18.581199 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.574329    2566 server.go:87] "Starting device plugin registration server"
Nov 03 19:36:18.581199 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.574581    2566 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 19:36:18.581199 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.574590    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 19:36:18.581199 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.574696    2566 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 19:36:18.581199 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.574776    2566 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 19:36:18.581199 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.574785    2566 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 19:36:18.581199 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:18.575318    2566 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 19:36:18.581199 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:18.575498    2566 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-206.us-east-2.compute.internal\" not found"
Nov 03 19:36:18.581199 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.579846    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 19:36:18.581199 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.581176    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 19:36:18.581199 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.581198    2566 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 19:36:18.581199 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.581214    2566 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 19:36:18.583985 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:18.581247    2566 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 19:36:18.584746 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.584726    2566 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:36:18.674836 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.674740    2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:36:18.675746 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.675731    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-206.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:36:18.675827 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.675771    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-206.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:36:18.675827 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.675782    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-206.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:36:18.675827 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.675811    2566 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:36:18.681670 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.681650    2566 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal"]
Nov 03 19:36:18.681728 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.681710    2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:36:18.682502 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.682474    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-206.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:36:18.682502 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.682503    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-206.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:36:18.682617 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.682517    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-206.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:36:18.684584 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.684570    2566 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:36:18.684642 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:18.684592    2566 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-206.us-east-2.compute.internal\": node \"ip-10-0-143-206.us-east-2.compute.internal\" not found"
Nov 03 19:36:18.684844 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.684832    2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:36:18.685065 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.685052    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:36:18.685101 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.685084    2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:36:18.685670 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.685654    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-206.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:36:18.685763 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.685686    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-206.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:36:18.685763 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.685702    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-206.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:36:18.685763 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.685713    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-206.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:36:18.685763 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.685740    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-206.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:36:18.685763 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.685753    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-206.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:36:18.688035 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.688016    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:36:18.688157 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.688049    2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:36:18.688829 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.688813    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-206.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:36:18.688918 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.688848    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-206.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:36:18.688918 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.688862    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-206.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:36:18.702002 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.701979    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6c67d5fc48584d861d9d388770293f30-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal\" (UID: \"6c67d5fc48584d861d9d388770293f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:36:18.702150 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.702007    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c67d5fc48584d861d9d388770293f30-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal\" (UID: \"6c67d5fc48584d861d9d388770293f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:36:18.702150 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.702028    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3def2722c7c58bcddedb41ddbc48c3ab-config\") pod \"kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal\" (UID: \"3def2722c7c58bcddedb41ddbc48c3ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:36:18.702276 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:18.702261    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-206.us-east-2.compute.internal\" not found"
Nov 03 19:36:18.707679 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:18.707660    2566 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-206.us-east-2.compute.internal\" not found" node="ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:36:18.709529 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:18.709513    2566 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-206.us-east-2.compute.internal\" not found" node="ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:36:18.802779 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:18.802746    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-206.us-east-2.compute.internal\" not found"
Nov 03 19:36:18.802779 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.802779    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6c67d5fc48584d861d9d388770293f30-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal\" (UID: \"6c67d5fc48584d861d9d388770293f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:36:18.803006 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.802805    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c67d5fc48584d861d9d388770293f30-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal\" (UID: \"6c67d5fc48584d861d9d388770293f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:36:18.803006 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.802830    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3def2722c7c58bcddedb41ddbc48c3ab-config\") pod \"kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal\" (UID: \"3def2722c7c58bcddedb41ddbc48c3ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:36:18.803006 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.802863    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3def2722c7c58bcddedb41ddbc48c3ab-config\") pod \"kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal\" (UID: \"3def2722c7c58bcddedb41ddbc48c3ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:36:18.803006 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.802871    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6c67d5fc48584d861d9d388770293f30-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal\" (UID: \"6c67d5fc48584d861d9d388770293f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:36:18.803006 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:18.802873    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c67d5fc48584d861d9d388770293f30-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal\" (UID: \"6c67d5fc48584d861d9d388770293f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:36:18.903046 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:18.903009    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-206.us-east-2.compute.internal\" not found"
Nov 03 19:36:19.003509 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:19.003433    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-206.us-east-2.compute.internal\" not found"
Nov 03 19:36:19.012595 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:19.012569    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:36:19.014728 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:19.014709    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:36:19.103753 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:19.103714    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-206.us-east-2.compute.internal\" not found"
Nov 03 19:36:19.204096 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:19.204067    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-206.us-east-2.compute.internal\" not found"
Nov 03 19:36:19.304542 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:19.304483    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-206.us-east-2.compute.internal\" not found"
Nov 03 19:36:19.392110 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:19.392077    2566 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:36:19.405212 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:19.405174    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-206.us-east-2.compute.internal\" not found"
Nov 03 19:36:19.425384 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:19.425352    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 19:36:19.425546 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:19.425497    2566 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 19:36:19.425546 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:19.425530    2566 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 19:36:19.425625 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:19.425533    2566 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 19:36:19.505689 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:19.505652    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-206.us-east-2.compute.internal\" not found"
Nov 03 19:36:19.513428 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:19.513366    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:31:19 +0000 UTC" deadline="2028-05-30 18:55:52.668160296 +0000 UTC"
Nov 03 19:36:19.513428 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:19.513427    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13775h19m33.154738872s"
Nov 03 19:36:19.605932 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:19.605854    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-206.us-east-2.compute.internal\" not found"
Nov 03 19:36:19.678100 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:19.678056    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3def2722c7c58bcddedb41ddbc48c3ab.slice/crio-e3b53f7107aff24d23d341b7d5189301d877d896ed6e3fd7f52eed5a7b978636 WatchSource:0}: Error finding container e3b53f7107aff24d23d341b7d5189301d877d896ed6e3fd7f52eed5a7b978636: Status 404 returned error can't find the container with id e3b53f7107aff24d23d341b7d5189301d877d896ed6e3fd7f52eed5a7b978636
Nov 03 19:36:19.678566 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:19.678544    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c67d5fc48584d861d9d388770293f30.slice/crio-ca5a374ce20ae82ec77cdcd80570f06fe84262dbb9729cd8a71a5c77f1bf9373 WatchSource:0}: Error finding container ca5a374ce20ae82ec77cdcd80570f06fe84262dbb9729cd8a71a5c77f1bf9373: Status 404 returned error can't find the container with id ca5a374ce20ae82ec77cdcd80570f06fe84262dbb9729cd8a71a5c77f1bf9373
Nov 03 19:36:19.683049 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:19.683020    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:36:19.706979 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:19.706948    2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-206.us-east-2.compute.internal\" not found"
Nov 03 19:36:19.752299 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:19.752268    2566 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:36:19.800390 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:19.800350    2566 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:36:19.813454 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:19.813423    2566 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:36:19.814629 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:19.814611    2566 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:36:19.824146 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:19.824108    2566 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:36:21.731102 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.731063    2566 apiserver.go:52] "Watching apiserver"
Nov 03 19:36:21.742774 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.742739    2566 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 19:36:21.743107 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.743078    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-q4tbd","openshift-multus/multus-xtsbd","kube-system/global-pull-secret-syncer-6q5vq","kube-system/konnectivity-agent-vh26d","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf","kube-system/kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-794nj","openshift-image-registry/node-ca-pswxd","openshift-network-diagnostics/network-check-target-pfcnb","openshift-network-operator/iptables-alerter-htzs4","openshift-ovn-kubernetes/ovnkube-node-x88xn","openshift-cluster-node-tuning-operator/tuned-mgrnm","openshift-dns/node-resolver-rzn4t","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal"]
Nov 03 19:36:21.745858 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.745823    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:21.745976 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:21.745924    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-794nj" podUID="ca39ebc9-da71-43c2-893f-c033684c9399"
Nov 03 19:36:21.747882 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.747859    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.751137 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.750026    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rzn4t"
Nov 03 19:36:21.752397 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.752376    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.753173 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.753156    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 19:36:21.753487 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.753467    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 19:36:21.753826 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.753809    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 19:36:21.753972 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.753957    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 19:36:21.754824 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.754803    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:21.754914 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:21.754873    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfcnb" podUID="f951b540-0179-474c-8a2c-7ee804f9d630"
Nov 03 19:36:21.755049 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.755033    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ndfpg\""
Nov 03 19:36:21.755119 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.755103    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qhpjg\""
Nov 03 19:36:21.757314 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.757295    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vh26d"
Nov 03 19:36:21.757734 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.757715    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 19:36:21.757734 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.757725    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lrbp4\""
Nov 03 19:36:21.760288 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.760266    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pswxd"
Nov 03 19:36:21.761654 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.761635    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 19:36:21.762056 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.762036    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:36:21.764664 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.764641    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-htzs4"
Nov 03 19:36:21.767116 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.767094    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:36:21.770414 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.770392    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.773029 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.773010    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:21.773132 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:21.773098    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6q5vq" podUID="98d32fbe-5011-47a8-b937-725a8cad95a4"
Nov 03 19:36:21.774023 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.774001    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/630922bd-4efc-4b44-96d1-fa55bce60e21-hosts-file\") pod \"node-resolver-rzn4t\" (UID: \"630922bd-4efc-4b44-96d1-fa55bce60e21\") " pod="openshift-dns/node-resolver-rzn4t"
Nov 03 19:36:21.774023 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.774013    2566 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 19:36:21.774156 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.774033    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-run-systemd\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.774156 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.774050    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.774272 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.774172    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-sys-fs\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:36:21.774272 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.774246    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-run-netns\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.774354 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.774282    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-multus-conf-dir\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.774354 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.774320    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-etc-selinux\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:36:21.774425 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.774379    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-etc-kubernetes\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.774425 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.774411    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-cni-bin\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.774519 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.774433    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttk6z\" (UniqueName: \"kubernetes.io/projected/facedcd2-6b70-4a3b-980c-04dceba2ef0d-kube-api-access-ttk6z\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:36:21.774519 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.774478    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/65b590ed-e1a5-467f-9283-7c875dd9ee24-ovnkube-script-lib\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.774519 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.774508    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b1cda986-f386-462d-9956-acc2a4de6e32-konnectivity-ca\") pod \"konnectivity-agent-vh26d\" (UID: \"b1cda986-f386-462d-9956-acc2a4de6e32\") " pod="kube-system/konnectivity-agent-vh26d"
Nov 03 19:36:21.774654 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.774572    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-socket-dir\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:36:21.774654 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.774607    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-multus-socket-dir-parent\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.774654 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.774644    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-slash\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.774764 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.774685    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-run-netns\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.774764 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.774723    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-run-ovn\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.774764 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.774742    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-node-log\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.774881 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.774768    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-cnibin\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.774881 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.774791    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jktld\" (UniqueName: \"kubernetes.io/projected/630922bd-4efc-4b44-96d1-fa55bce60e21-kube-api-access-jktld\") pod \"node-resolver-rzn4t\" (UID: \"630922bd-4efc-4b44-96d1-fa55bce60e21\") " pod="openshift-dns/node-resolver-rzn4t"
Nov 03 19:36:21.774881 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.774826    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/65b590ed-e1a5-467f-9283-7c875dd9ee24-ovnkube-config\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.774881 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.774866    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/65b590ed-e1a5-467f-9283-7c875dd9ee24-ovn-node-metrics-cert\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.775100 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.774896    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-hostroot\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.775100 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.774918    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:36:21.775100 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.774943    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-registration-dir\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:36:21.775100 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775007    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpc4k\" (UniqueName: \"kubernetes.io/projected/ca39ebc9-da71-43c2-893f-c033684c9399-kube-api-access-gpc4k\") pod \"network-metrics-daemon-794nj\" (UID: \"ca39ebc9-da71-43c2-893f-c033684c9399\") " pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:21.775100 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775065    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-multus-cni-dir\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.775100 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775089    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-run-k8s-cni-cncf-io\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.775426 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775127    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nx7w\" (UniqueName: \"kubernetes.io/projected/88a27c5e-de31-4090-be4d-6b8d6551e4c7-kube-api-access-5nx7w\") pod \"node-ca-pswxd\" (UID: \"88a27c5e-de31-4090-be4d-6b8d6551e4c7\") " pod="openshift-image-registry/node-ca-pswxd"
Nov 03 19:36:21.775426 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775142    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/055215b4-9de9-494c-bbff-288cc97baf2f-host-slash\") pod \"iptables-alerter-htzs4\" (UID: \"055215b4-9de9-494c-bbff-288cc97baf2f\") " pod="openshift-network-operator/iptables-alerter-htzs4"
Nov 03 19:36:21.775426 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775156    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-system-cni-dir\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.775426 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775171    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-var-lib-cni-bin\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.775426 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775186    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-etc-openvswitch\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.775426 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775225    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-log-socket\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.775426 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775281    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-run-ovn-kubernetes\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.775426 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775317    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b1cda986-f386-462d-9956-acc2a4de6e32-agent-certs\") pod \"konnectivity-agent-vh26d\" (UID: \"b1cda986-f386-462d-9956-acc2a4de6e32\") " pod="kube-system/konnectivity-agent-vh26d"
Nov 03 19:36:21.775426 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775357    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-var-lib-openvswitch\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.775426 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775384    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/65b590ed-e1a5-467f-9283-7c875dd9ee24-env-overrides\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.775426 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775423    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-kubelet\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.775924 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775454    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/88a27c5e-de31-4090-be4d-6b8d6551e4c7-serviceca\") pod \"node-ca-pswxd\" (UID: \"88a27c5e-de31-4090-be4d-6b8d6551e4c7\") " pod="openshift-image-registry/node-ca-pswxd"
Nov 03 19:36:21.775924 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775499    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-cni-binary-copy\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.775924 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775525    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-run-multus-certs\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.775924 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775562    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/630922bd-4efc-4b44-96d1-fa55bce60e21-tmp-dir\") pod \"node-resolver-rzn4t\" (UID: \"630922bd-4efc-4b44-96d1-fa55bce60e21\") " pod="openshift-dns/node-resolver-rzn4t"
Nov 03 19:36:21.775924 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775595    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88a27c5e-de31-4090-be4d-6b8d6551e4c7-host\") pod \"node-ca-pswxd\" (UID: \"88a27c5e-de31-4090-be4d-6b8d6551e4c7\") " pod="openshift-image-registry/node-ca-pswxd"
Nov 03 19:36:21.775924 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775627    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca39ebc9-da71-43c2-893f-c033684c9399-metrics-certs\") pod \"network-metrics-daemon-794nj\" (UID: \"ca39ebc9-da71-43c2-893f-c033684c9399\") " pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:21.775924 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775680    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-multus-daemon-config\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.775924 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775715    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgxw5\" (UniqueName: \"kubernetes.io/projected/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-kube-api-access-dgxw5\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.775924 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775748    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-var-lib-kubelet\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.775924 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775777    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-systemd-units\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.775924 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775799    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66pfk\" (UniqueName: \"kubernetes.io/projected/055215b4-9de9-494c-bbff-288cc97baf2f-kube-api-access-66pfk\") pod \"iptables-alerter-htzs4\" (UID: \"055215b4-9de9-494c-bbff-288cc97baf2f\") " pod="openshift-network-operator/iptables-alerter-htzs4"
Nov 03 19:36:21.775924 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775832    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-run-openvswitch\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.775924 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775867    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-cni-netd\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.775924 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775896    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-var-lib-cni-multus\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.775924 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775918    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgdxj\" (UniqueName: \"kubernetes.io/projected/65b590ed-e1a5-467f-9283-7c875dd9ee24-kube-api-access-vgdxj\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.776611 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775940    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-device-dir\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:36:21.776611 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775977    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng4gr\" (UniqueName: \"kubernetes.io/projected/f951b540-0179-474c-8a2c-7ee804f9d630-kube-api-access-ng4gr\") pod \"network-check-target-pfcnb\" (UID: \"f951b540-0179-474c-8a2c-7ee804f9d630\") " pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:21.776611 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.775998    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/055215b4-9de9-494c-bbff-288cc97baf2f-iptables-alerter-script\") pod \"iptables-alerter-htzs4\" (UID: \"055215b4-9de9-494c-bbff-288cc97baf2f\") " pod="openshift-network-operator/iptables-alerter-htzs4"
Nov 03 19:36:21.792002 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.791975    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qh8nr"
Nov 03 19:36:21.804571 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.804542    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qh8nr"
Nov 03 19:36:21.849875 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.849829    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal" event={"ID":"6c67d5fc48584d861d9d388770293f30","Type":"ContainerStarted","Data":"ca5a374ce20ae82ec77cdcd80570f06fe84262dbb9729cd8a71a5c77f1bf9373"}
Nov 03 19:36:21.851483 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.851452    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal" event={"ID":"3def2722c7c58bcddedb41ddbc48c3ab","Type":"ContainerStarted","Data":"e3b53f7107aff24d23d341b7d5189301d877d896ed6e3fd7f52eed5a7b978636"}
Nov 03 19:36:21.853784 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.853717    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 19:36:21.853784 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.853728    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 19:36:21.853784 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.853741    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 19:36:21.853784 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.853769    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 19:36:21.853995 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.853797    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 19:36:21.854920 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.854900    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:36:21.855004 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.854967    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 19:36:21.855004 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.854969    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 19:36:21.855107 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.855012    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 19:36:21.855473 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.855285    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:36:21.855473 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.855309    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 19:36:21.855473 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.855319    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 19:36:21.855473 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.855344    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 19:36:21.855473 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.855366    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 19:36:21.855473 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.855401    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 19:36:21.856029 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.856008    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 19:36:21.856098 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.856073    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 19:36:21.856157 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.856136    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 19:36:21.856623 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.856600    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 19:36:21.857220 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.857183    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jzp4r\""
Nov 03 19:36:21.857319 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.857238    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ldllm\""
Nov 03 19:36:21.857396 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.857377    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 19:36:21.857519 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.857503    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xbkg2\""
Nov 03 19:36:21.857588 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.857378    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 19:36:21.857673 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.857652    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xv4gr\""
Nov 03 19:36:21.857723 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.857677    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rs44g\""
Nov 03 19:36:21.857904 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.857887    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bkd9f\""
Nov 03 19:36:21.859406 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.856198    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 19:36:21.863575 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.863557    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 19:36:21.876486 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876195    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-var-lib-kubelet\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.876486 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876265    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-systemd-units\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.876486 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876284    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66pfk\" (UniqueName: \"kubernetes.io/projected/055215b4-9de9-494c-bbff-288cc97baf2f-kube-api-access-66pfk\") pod \"iptables-alerter-htzs4\" (UID: \"055215b4-9de9-494c-bbff-288cc97baf2f\") " pod="openshift-network-operator/iptables-alerter-htzs4"
Nov 03 19:36:21.876486 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876298    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-var-lib-kubelet\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.876486 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876305    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4a42db6-0377-4a5a-94ab-be3f82b332aa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:36:21.876486 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876471    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-systemd-units\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.876859 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876508    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-run-openvswitch\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.876859 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876529    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-cni-netd\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.876859 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876558    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-sys\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.876859 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876581    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-lib-modules\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.876859 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876605    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-var-lib-cni-multus\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.876859 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876604    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-run-openvswitch\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.876859 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876621    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-cni-netd\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.876859 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876658    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgdxj\" (UniqueName: \"kubernetes.io/projected/65b590ed-e1a5-467f-9283-7c875dd9ee24-kube-api-access-vgdxj\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.876859 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876664    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-var-lib-cni-multus\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.876859 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876692    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-device-dir\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:36:21.876859 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876719    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-tuned\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.876859 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876734    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-device-dir\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:36:21.876859 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876737    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng4gr\" (UniqueName: \"kubernetes.io/projected/f951b540-0179-474c-8a2c-7ee804f9d630-kube-api-access-ng4gr\") pod \"network-check-target-pfcnb\" (UID: \"f951b540-0179-474c-8a2c-7ee804f9d630\") " pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:21.876859 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876760    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/055215b4-9de9-494c-bbff-288cc97baf2f-iptables-alerter-script\") pod \"iptables-alerter-htzs4\" (UID: \"055215b4-9de9-494c-bbff-288cc97baf2f\") " pod="openshift-network-operator/iptables-alerter-htzs4"
Nov 03 19:36:21.876859 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876784    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4a42db6-0377-4a5a-94ab-be3f82b332aa-cnibin\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:36:21.876859 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876799    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-systemd\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.876859 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876851    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-host\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.877381 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876876    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/630922bd-4efc-4b44-96d1-fa55bce60e21-hosts-file\") pod \"node-resolver-rzn4t\" (UID: \"630922bd-4efc-4b44-96d1-fa55bce60e21\") " pod="openshift-dns/node-resolver-rzn4t"
Nov 03 19:36:21.877381 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876891    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-run-systemd\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.877381 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876910    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.877381 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876945    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-run-systemd\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.877381 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876975    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/630922bd-4efc-4b44-96d1-fa55bce60e21-hosts-file\") pod \"node-resolver-rzn4t\" (UID: \"630922bd-4efc-4b44-96d1-fa55bce60e21\") " pod="openshift-dns/node-resolver-rzn4t"
Nov 03 19:36:21.877381 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.876980    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.877381 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877024    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-sys-fs\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:36:21.877381 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877050    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-run\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.877381 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877085    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-run-netns\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.877381 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877112    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-sys-fs\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:36:21.877381 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877109    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-multus-conf-dir\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.877381 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877136    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-etc-selinux\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:36:21.877381 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877150    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-multus-conf-dir\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.877381 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877152    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-sysctl-conf\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.877381 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877217    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-run-netns\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.877381 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877227    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/98d32fbe-5011-47a8-b937-725a8cad95a4-dbus\") pod \"global-pull-secret-syncer-6q5vq\" (UID: \"98d32fbe-5011-47a8-b937-725a8cad95a4\") " pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:21.877381 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877255    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-etc-kubernetes\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.877836 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877279    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-cni-bin\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.877836 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877295    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-etc-selinux\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:36:21.877836 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877302    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttk6z\" (UniqueName: \"kubernetes.io/projected/facedcd2-6b70-4a3b-980c-04dceba2ef0d-kube-api-access-ttk6z\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:36:21.877836 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877322    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-etc-kubernetes\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.877836 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877330    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b4a42db6-0377-4a5a-94ab-be3f82b332aa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:36:21.877836 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877354    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-cni-bin\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.877836 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877359    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzmv5\" (UniqueName: \"kubernetes.io/projected/b4a42db6-0377-4a5a-94ab-be3f82b332aa-kube-api-access-tzmv5\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:36:21.877836 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877387    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/65b590ed-e1a5-467f-9283-7c875dd9ee24-ovnkube-script-lib\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.877836 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877387    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/055215b4-9de9-494c-bbff-288cc97baf2f-iptables-alerter-script\") pod \"iptables-alerter-htzs4\" (UID: \"055215b4-9de9-494c-bbff-288cc97baf2f\") " pod="openshift-network-operator/iptables-alerter-htzs4"
Nov 03 19:36:21.877836 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877410    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b1cda986-f386-462d-9956-acc2a4de6e32-konnectivity-ca\") pod \"konnectivity-agent-vh26d\" (UID: \"b1cda986-f386-462d-9956-acc2a4de6e32\") " pod="kube-system/konnectivity-agent-vh26d"
Nov 03 19:36:21.877836 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877435    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-socket-dir\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:36:21.877836 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877543    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-socket-dir\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:36:21.877836 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877572    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-multus-socket-dir-parent\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.877836 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877596    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-slash\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.877836 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877620    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-run-netns\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.877836 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877643    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-run-ovn\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.877836 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877673    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-node-log\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.878290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877696    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-cnibin\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.878290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877734    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jktld\" (UniqueName: \"kubernetes.io/projected/630922bd-4efc-4b44-96d1-fa55bce60e21-kube-api-access-jktld\") pod \"node-resolver-rzn4t\" (UID: \"630922bd-4efc-4b44-96d1-fa55bce60e21\") " pod="openshift-dns/node-resolver-rzn4t"
Nov 03 19:36:21.878290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877758    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/65b590ed-e1a5-467f-9283-7c875dd9ee24-ovnkube-config\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.878290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877783    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/65b590ed-e1a5-467f-9283-7c875dd9ee24-ovn-node-metrics-cert\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.878290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877808    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-modprobe-d\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.878290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877832    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-sysconfig\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.878290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877855    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71c679be-e63f-4ad4-a1b2-5474ec4fa125-tmp\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.878290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877877    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-hostroot\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.878290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877901    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:36:21.878290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877910    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-cnibin\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.878290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877925    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-registration-dir\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:36:21.878290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877951    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zmfm\" (UniqueName: \"kubernetes.io/projected/71c679be-e63f-4ad4-a1b2-5474ec4fa125-kube-api-access-4zmfm\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.878290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877975    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98d32fbe-5011-47a8-b937-725a8cad95a4-original-pull-secret\") pod \"global-pull-secret-syncer-6q5vq\" (UID: \"98d32fbe-5011-47a8-b937-725a8cad95a4\") " pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:21.878290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877985    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/65b590ed-e1a5-467f-9283-7c875dd9ee24-ovnkube-script-lib\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.878290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877989    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-multus-socket-dir-parent\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.878290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878017    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpc4k\" (UniqueName: \"kubernetes.io/projected/ca39ebc9-da71-43c2-893f-c033684c9399-kube-api-access-gpc4k\") pod \"network-metrics-daemon-794nj\" (UID: \"ca39ebc9-da71-43c2-893f-c033684c9399\") " pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:21.878290 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878035    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-slash\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.878732 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878046    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-multus-cni-dir\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.878732 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878056    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-hostroot\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.878732 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878063    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-run-netns\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.878732 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878093    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-run-ovn\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.878732 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878096    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-run-k8s-cni-cncf-io\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.878732 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878121    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nx7w\" (UniqueName: \"kubernetes.io/projected/88a27c5e-de31-4090-be4d-6b8d6551e4c7-kube-api-access-5nx7w\") pod \"node-ca-pswxd\" (UID: \"88a27c5e-de31-4090-be4d-6b8d6551e4c7\") " pod="openshift-image-registry/node-ca-pswxd"
Nov 03 19:36:21.878732 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878129    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-node-log\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.878732 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878147    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/055215b4-9de9-494c-bbff-288cc97baf2f-host-slash\") pod \"iptables-alerter-htzs4\" (UID: \"055215b4-9de9-494c-bbff-288cc97baf2f\") " pod="openshift-network-operator/iptables-alerter-htzs4"
Nov 03 19:36:21.878732 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878163    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-system-cni-dir\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.878732 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878179    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-var-lib-cni-bin\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.878732 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878193    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-etc-openvswitch\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.878732 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878226    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-log-socket\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.878732 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878242    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-run-ovn-kubernetes\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.878732 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878256    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b1cda986-f386-462d-9956-acc2a4de6e32-agent-certs\") pod \"konnectivity-agent-vh26d\" (UID: \"b1cda986-f386-462d-9956-acc2a4de6e32\") " pod="kube-system/konnectivity-agent-vh26d"
Nov 03 19:36:21.878732 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878271    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-var-lib-openvswitch\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.878732 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878285    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/65b590ed-e1a5-467f-9283-7c875dd9ee24-env-overrides\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.878732 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878301    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4a42db6-0377-4a5a-94ab-be3f82b332aa-system-cni-dir\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:36:21.878732 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878317    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-sysctl-d\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.879260 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.877878    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b1cda986-f386-462d-9956-acc2a4de6e32-konnectivity-ca\") pod \"konnectivity-agent-vh26d\" (UID: \"b1cda986-f386-462d-9956-acc2a4de6e32\") " pod="kube-system/konnectivity-agent-vh26d"
Nov 03 19:36:21.879260 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878332    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-kubelet\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.879260 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878349    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-kubelet\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.879260 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878375    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-var-lib-cni-bin\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.879260 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878394    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-etc-openvswitch\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.879260 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878413    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-log-socket\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.879260 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878433    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-run-ovn-kubernetes\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.879260 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878398    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/88a27c5e-de31-4090-be4d-6b8d6551e4c7-serviceca\") pod \"node-ca-pswxd\" (UID: \"88a27c5e-de31-4090-be4d-6b8d6551e4c7\") " pod="openshift-image-registry/node-ca-pswxd"
Nov 03 19:36:21.879260 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878468    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-kubernetes\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.879260 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878498    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-cni-binary-copy\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.879260 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878525    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/65b590ed-e1a5-467f-9283-7c875dd9ee24-ovnkube-config\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.879260 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878525    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-run-multus-certs\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.879260 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878567    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-run-multus-certs\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.879260 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878571    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/630922bd-4efc-4b44-96d1-fa55bce60e21-tmp-dir\") pod \"node-resolver-rzn4t\" (UID: \"630922bd-4efc-4b44-96d1-fa55bce60e21\") " pod="openshift-dns/node-resolver-rzn4t"
Nov 03 19:36:21.879260 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878599    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88a27c5e-de31-4090-be4d-6b8d6551e4c7-host\") pod \"node-ca-pswxd\" (UID: \"88a27c5e-de31-4090-be4d-6b8d6551e4c7\") " pod="openshift-image-registry/node-ca-pswxd"
Nov 03 19:36:21.879260 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878628    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4a42db6-0377-4a5a-94ab-be3f82b332aa-cni-binary-copy\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:36:21.879260 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878653    2566 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 19:36:21.879260 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878682    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:36:21.879749 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878808    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/630922bd-4efc-4b44-96d1-fa55bce60e21-tmp-dir\") pod \"node-resolver-rzn4t\" (UID: \"630922bd-4efc-4b44-96d1-fa55bce60e21\") " pod="openshift-dns/node-resolver-rzn4t"
Nov 03 19:36:21.879749 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.878904    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/055215b4-9de9-494c-bbff-288cc97baf2f-host-slash\") pod \"iptables-alerter-htzs4\" (UID: \"055215b4-9de9-494c-bbff-288cc97baf2f\") " pod="openshift-network-operator/iptables-alerter-htzs4"
Nov 03 19:36:21.879749 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.879043    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b4a42db6-0377-4a5a-94ab-be3f82b332aa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:36:21.879749 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.879070    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88a27c5e-de31-4090-be4d-6b8d6551e4c7-host\") pod \"node-ca-pswxd\" (UID: \"88a27c5e-de31-4090-be4d-6b8d6551e4c7\") " pod="openshift-image-registry/node-ca-pswxd"
Nov 03 19:36:21.879749 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.879084    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-var-lib-kubelet\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.879749 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.879101    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-var-lib-openvswitch\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.879749 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.879114    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca39ebc9-da71-43c2-893f-c033684c9399-metrics-certs\") pod \"network-metrics-daemon-794nj\" (UID: \"ca39ebc9-da71-43c2-893f-c033684c9399\") " pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:21.879749 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.879140    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-multus-daemon-config\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.879749 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.879165    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgxw5\" (UniqueName: \"kubernetes.io/projected/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-kube-api-access-dgxw5\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.879749 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.879216    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/98d32fbe-5011-47a8-b937-725a8cad95a4-kubelet-config\") pod \"global-pull-secret-syncer-6q5vq\" (UID: \"98d32fbe-5011-47a8-b937-725a8cad95a4\") " pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:21.879749 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.879309    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-registration-dir\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:36:21.879749 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.879319    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/65b590ed-e1a5-467f-9283-7c875dd9ee24-env-overrides\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.879749 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.879389    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-system-cni-dir\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.879749 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.879546    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-multus-cni-dir\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.879749 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.879579    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-cni-binary-copy\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.879749 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.879586    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-run-k8s-cni-cncf-io\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.879749 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:21.879669    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:21.880277 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.879754    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/88a27c5e-de31-4090-be4d-6b8d6551e4c7-serviceca\") pod \"node-ca-pswxd\" (UID: \"88a27c5e-de31-4090-be4d-6b8d6551e4c7\") " pod="openshift-image-registry/node-ca-pswxd"
Nov 03 19:36:21.880277 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:21.880028    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca39ebc9-da71-43c2-893f-c033684c9399-metrics-certs podName:ca39ebc9-da71-43c2-893f-c033684c9399 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:22.379999104 +0000 UTC m=+3.025189408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca39ebc9-da71-43c2-893f-c033684c9399-metrics-certs") pod "network-metrics-daemon-794nj" (UID: "ca39ebc9-da71-43c2-893f-c033684c9399") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:21.880747 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.880725    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-multus-daemon-config\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.882647 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.882613    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/65b590ed-e1a5-467f-9283-7c875dd9ee24-ovn-node-metrics-cert\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.882647 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.882618    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b1cda986-f386-462d-9956-acc2a4de6e32-agent-certs\") pod \"konnectivity-agent-vh26d\" (UID: \"b1cda986-f386-462d-9956-acc2a4de6e32\") " pod="kube-system/konnectivity-agent-vh26d"
Nov 03 19:36:21.893272 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:21.893246    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:36:21.893272 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:21.893272    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:36:21.893433 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:21.893286    2566 projected.go:196] Error preparing data for projected volume kube-api-access-ng4gr for pod openshift-network-diagnostics/network-check-target-pfcnb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:21.893433 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:21.893362    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f951b540-0179-474c-8a2c-7ee804f9d630-kube-api-access-ng4gr podName:f951b540-0179-474c-8a2c-7ee804f9d630 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:22.393343558 +0000 UTC m=+3.038533870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ng4gr" (UniqueName: "kubernetes.io/projected/f951b540-0179-474c-8a2c-7ee804f9d630-kube-api-access-ng4gr") pod "network-check-target-pfcnb" (UID: "f951b540-0179-474c-8a2c-7ee804f9d630") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:21.895178 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.895149    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66pfk\" (UniqueName: \"kubernetes.io/projected/055215b4-9de9-494c-bbff-288cc97baf2f-kube-api-access-66pfk\") pod \"iptables-alerter-htzs4\" (UID: \"055215b4-9de9-494c-bbff-288cc97baf2f\") " pod="openshift-network-operator/iptables-alerter-htzs4"
Nov 03 19:36:21.896703 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.896675    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nx7w\" (UniqueName: \"kubernetes.io/projected/88a27c5e-de31-4090-be4d-6b8d6551e4c7-kube-api-access-5nx7w\") pod \"node-ca-pswxd\" (UID: \"88a27c5e-de31-4090-be4d-6b8d6551e4c7\") " pod="openshift-image-registry/node-ca-pswxd"
Nov 03 19:36:21.896926 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.896898    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpc4k\" (UniqueName: \"kubernetes.io/projected/ca39ebc9-da71-43c2-893f-c033684c9399-kube-api-access-gpc4k\") pod \"network-metrics-daemon-794nj\" (UID: \"ca39ebc9-da71-43c2-893f-c033684c9399\") " pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:21.897009 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.896929    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgdxj\" (UniqueName: \"kubernetes.io/projected/65b590ed-e1a5-467f-9283-7c875dd9ee24-kube-api-access-vgdxj\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:21.897241 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.897198    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttk6z\" (UniqueName: \"kubernetes.io/projected/facedcd2-6b70-4a3b-980c-04dceba2ef0d-kube-api-access-ttk6z\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:36:21.897449 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.897423    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jktld\" (UniqueName: \"kubernetes.io/projected/630922bd-4efc-4b44-96d1-fa55bce60e21-kube-api-access-jktld\") pod \"node-resolver-rzn4t\" (UID: \"630922bd-4efc-4b44-96d1-fa55bce60e21\") " pod="openshift-dns/node-resolver-rzn4t"
Nov 03 19:36:21.898510 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.898489    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgxw5\" (UniqueName: \"kubernetes.io/projected/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-kube-api-access-dgxw5\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:21.927627 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.927594    2566 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:36:21.980275 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.980231    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98d32fbe-5011-47a8-b937-725a8cad95a4-original-pull-secret\") pod \"global-pull-secret-syncer-6q5vq\" (UID: \"98d32fbe-5011-47a8-b937-725a8cad95a4\") " pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:21.980458 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.980294    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4a42db6-0377-4a5a-94ab-be3f82b332aa-system-cni-dir\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:36:21.980458 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.980322    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-sysctl-d\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.980458 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.980350    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-kubernetes\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.980458 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.980381    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4a42db6-0377-4a5a-94ab-be3f82b332aa-cni-binary-copy\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:36:21.980458 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.980406    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b4a42db6-0377-4a5a-94ab-be3f82b332aa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:36:21.980458 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.980434    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-var-lib-kubelet\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.980704 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.980472    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/98d32fbe-5011-47a8-b937-725a8cad95a4-kubelet-config\") pod \"global-pull-secret-syncer-6q5vq\" (UID: \"98d32fbe-5011-47a8-b937-725a8cad95a4\") " pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:21.980704 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.980500    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4a42db6-0377-4a5a-94ab-be3f82b332aa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:36:21.980704 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.980523    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-sys\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.980704 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.980544    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-lib-modules\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.980704 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.980570    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-tuned\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.980704 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.980604    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4a42db6-0377-4a5a-94ab-be3f82b332aa-cnibin\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:36:21.980704 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.980626    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-systemd\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.980704 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.980647    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-host\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.980704 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.980673    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-run\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.980704 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.980699    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-sysctl-conf\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.981034 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.980723    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/98d32fbe-5011-47a8-b937-725a8cad95a4-dbus\") pod \"global-pull-secret-syncer-6q5vq\" (UID: \"98d32fbe-5011-47a8-b937-725a8cad95a4\") " pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:21.981034 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.980750    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b4a42db6-0377-4a5a-94ab-be3f82b332aa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:36:21.981034 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.980776    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzmv5\" (UniqueName: \"kubernetes.io/projected/b4a42db6-0377-4a5a-94ab-be3f82b332aa-kube-api-access-tzmv5\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:36:21.981034 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.980811    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-modprobe-d\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.981034 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.980833    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-sysconfig\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.981034 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.980856    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71c679be-e63f-4ad4-a1b2-5474ec4fa125-tmp\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.981034 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.980883    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zmfm\" (UniqueName: \"kubernetes.io/projected/71c679be-e63f-4ad4-a1b2-5474ec4fa125-kube-api-access-4zmfm\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.981344 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:21.981327    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:21.981413 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:21.981403    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d32fbe-5011-47a8-b937-725a8cad95a4-original-pull-secret podName:98d32fbe-5011-47a8-b937-725a8cad95a4 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:22.481384501 +0000 UTC m=+3.126574803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/98d32fbe-5011-47a8-b937-725a8cad95a4-original-pull-secret") pod "global-pull-secret-syncer-6q5vq" (UID: "98d32fbe-5011-47a8-b937-725a8cad95a4") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:21.981577 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.981563    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4a42db6-0377-4a5a-94ab-be3f82b332aa-system-cni-dir\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:36:21.981678 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.981666    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-sysctl-d\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.981719 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.981700    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-kubernetes\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.982160 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.982137    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4a42db6-0377-4a5a-94ab-be3f82b332aa-cni-binary-copy\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:36:21.982331 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.982312    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-host\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.982408 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.982374    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-var-lib-kubelet\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.982449 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.982426    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/98d32fbe-5011-47a8-b937-725a8cad95a4-kubelet-config\") pod \"global-pull-secret-syncer-6q5vq\" (UID: \"98d32fbe-5011-47a8-b937-725a8cad95a4\") " pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:21.982582 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.982559    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4a42db6-0377-4a5a-94ab-be3f82b332aa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:36:21.982662 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.982623    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-sys\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.982662 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.982626    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b4a42db6-0377-4a5a-94ab-be3f82b332aa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:36:21.982755 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.982675    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-run\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.982755 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.982728    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-lib-modules\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.982833 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.982786    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-sysctl-conf\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.982878 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.982839    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/98d32fbe-5011-47a8-b937-725a8cad95a4-dbus\") pod \"global-pull-secret-syncer-6q5vq\" (UID: \"98d32fbe-5011-47a8-b937-725a8cad95a4\") " pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:21.983247 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.983228    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b4a42db6-0377-4a5a-94ab-be3f82b332aa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:36:21.983504 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.983482    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-modprobe-d\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.983588 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.983545    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-sysconfig\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.983904 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.983870    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4a42db6-0377-4a5a-94ab-be3f82b332aa-cnibin\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:36:21.984182 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.984162    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-systemd\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.985420 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.985379    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-tuned\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.985841 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.985823    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71c679be-e63f-4ad4-a1b2-5474ec4fa125-tmp\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.993935 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.993910    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zmfm\" (UniqueName: \"kubernetes.io/projected/71c679be-e63f-4ad4-a1b2-5474ec4fa125-kube-api-access-4zmfm\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:21.995097 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:21.995072    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzmv5\" (UniqueName: \"kubernetes.io/projected/b4a42db6-0377-4a5a-94ab-be3f82b332aa-kube-api-access-tzmv5\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:36:22.064100 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.064058    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xtsbd"
Nov 03 19:36:22.073669 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.073499    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rzn4t"
Nov 03 19:36:22.075989 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:22.075958    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32452ae6_08d2_4fb2_863d_72dc09d4c0a2.slice/crio-da480f56accb6891c3c0d00769c98771f2bd08c62e5ec777516d31555cb5a27d WatchSource:0}: Error finding container da480f56accb6891c3c0d00769c98771f2bd08c62e5ec777516d31555cb5a27d: Status 404 returned error can't find the container with id da480f56accb6891c3c0d00769c98771f2bd08c62e5ec777516d31555cb5a27d
Nov 03 19:36:22.084074 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:22.084045    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630922bd_4efc_4b44_96d1_fa55bce60e21.slice/crio-f11f7ea9334844e8aadb4a2b85717570432be27063013dd93e6846a27bc4bf14 WatchSource:0}: Error finding container f11f7ea9334844e8aadb4a2b85717570432be27063013dd93e6846a27bc4bf14: Status 404 returned error can't find the container with id f11f7ea9334844e8aadb4a2b85717570432be27063013dd93e6846a27bc4bf14
Nov 03 19:36:22.102462 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.102434    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vh26d"
Nov 03 19:36:22.103451 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.103428    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pswxd"
Nov 03 19:36:22.105749 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.105723    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:22.115886 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.115862    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:36:22.116439 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:22.116408    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b590ed_e1a5_467f_9283_7c875dd9ee24.slice/crio-dc002ae6adfc9db21f86304ef7cef337adb0de17d39be256b6634f686cf10d88 WatchSource:0}: Error finding container dc002ae6adfc9db21f86304ef7cef337adb0de17d39be256b6634f686cf10d88: Status 404 returned error can't find the container with id dc002ae6adfc9db21f86304ef7cef337adb0de17d39be256b6634f686cf10d88
Nov 03 19:36:22.117902 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:22.117867    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1cda986_f386_462d_9956_acc2a4de6e32.slice/crio-5a7f41c5228d6776608f44fd25798b14838c46bc7bdb1a5f798a7071d15e8c35 WatchSource:0}: Error finding container 5a7f41c5228d6776608f44fd25798b14838c46bc7bdb1a5f798a7071d15e8c35: Status 404 returned error can't find the container with id 5a7f41c5228d6776608f44fd25798b14838c46bc7bdb1a5f798a7071d15e8c35
Nov 03 19:36:22.122759 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.122710    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-htzs4"
Nov 03 19:36:22.124772 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:22.124745    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a27c5e_de31_4090_be4d_6b8d6551e4c7.slice/crio-e4d01dcd5adba2d4060227e841b164fc8265e16eec03d9d866e829e9e8e8af84 WatchSource:0}: Error finding container e4d01dcd5adba2d4060227e841b164fc8265e16eec03d9d866e829e9e8e8af84: Status 404 returned error can't find the container with id e4d01dcd5adba2d4060227e841b164fc8265e16eec03d9d866e829e9e8e8af84
Nov 03 19:36:22.135016 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:22.134976    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod055215b4_9de9_494c_bbff_288cc97baf2f.slice/crio-f45ff49a9cf6737e1a1324c65dd202171c2106d3b3d10b63706762b06b9b7cf4 WatchSource:0}: Error finding container f45ff49a9cf6737e1a1324c65dd202171c2106d3b3d10b63706762b06b9b7cf4: Status 404 returned error can't find the container with id f45ff49a9cf6737e1a1324c65dd202171c2106d3b3d10b63706762b06b9b7cf4
Nov 03 19:36:22.143409 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.143382    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:36:22.144860 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.144837    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:36:22.155552 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:22.155510    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c679be_e63f_4ad4_a1b2_5474ec4fa125.slice/crio-dcbdef7fc62a5ddb080912db6bee1f7c41b56171960aabe80dc6ab7257fc9264 WatchSource:0}: Error finding container dcbdef7fc62a5ddb080912db6bee1f7c41b56171960aabe80dc6ab7257fc9264: Status 404 returned error can't find the container with id dcbdef7fc62a5ddb080912db6bee1f7c41b56171960aabe80dc6ab7257fc9264
Nov 03 19:36:22.158851 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:36:22.158824    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a42db6_0377_4a5a_94ab_be3f82b332aa.slice/crio-8edd2f162297442f9f77e5ba98660f4c491934551e31553c054c9eecdd98d5b1 WatchSource:0}: Error finding container 8edd2f162297442f9f77e5ba98660f4c491934551e31553c054c9eecdd98d5b1: Status 404 returned error can't find the container with id 8edd2f162297442f9f77e5ba98660f4c491934551e31553c054c9eecdd98d5b1
Nov 03 19:36:22.244388 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.244357    2566 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:36:22.384608 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.384537    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca39ebc9-da71-43c2-893f-c033684c9399-metrics-certs\") pod \"network-metrics-daemon-794nj\" (UID: \"ca39ebc9-da71-43c2-893f-c033684c9399\") " pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:22.384753 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:22.384661    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:22.384753 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:22.384727    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca39ebc9-da71-43c2-893f-c033684c9399-metrics-certs podName:ca39ebc9-da71-43c2-893f-c033684c9399 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:23.384711925 +0000 UTC m=+4.029902207 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca39ebc9-da71-43c2-893f-c033684c9399-metrics-certs") pod "network-metrics-daemon-794nj" (UID: "ca39ebc9-da71-43c2-893f-c033684c9399") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:22.485412 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.485378    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng4gr\" (UniqueName: \"kubernetes.io/projected/f951b540-0179-474c-8a2c-7ee804f9d630-kube-api-access-ng4gr\") pod \"network-check-target-pfcnb\" (UID: \"f951b540-0179-474c-8a2c-7ee804f9d630\") " pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:22.485578 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.485457    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98d32fbe-5011-47a8-b937-725a8cad95a4-original-pull-secret\") pod \"global-pull-secret-syncer-6q5vq\" (UID: \"98d32fbe-5011-47a8-b937-725a8cad95a4\") " pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:22.485652 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:22.485567    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:36:22.485652 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:22.485608    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:36:22.485652 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:22.485624    2566 projected.go:196] Error preparing data for projected volume kube-api-access-ng4gr for pod openshift-network-diagnostics/network-check-target-pfcnb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:22.485802 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:22.485572    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:22.485802 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:22.485679    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f951b540-0179-474c-8a2c-7ee804f9d630-kube-api-access-ng4gr podName:f951b540-0179-474c-8a2c-7ee804f9d630 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:23.485660436 +0000 UTC m=+4.130850731 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ng4gr" (UniqueName: "kubernetes.io/projected/f951b540-0179-474c-8a2c-7ee804f9d630-kube-api-access-ng4gr") pod "network-check-target-pfcnb" (UID: "f951b540-0179-474c-8a2c-7ee804f9d630") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:22.485802 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:22.485708    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d32fbe-5011-47a8-b937-725a8cad95a4-original-pull-secret podName:98d32fbe-5011-47a8-b937-725a8cad95a4 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:23.485696563 +0000 UTC m=+4.130886858 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/98d32fbe-5011-47a8-b937-725a8cad95a4-original-pull-secret") pod "global-pull-secret-syncer-6q5vq" (UID: "98d32fbe-5011-47a8-b937-725a8cad95a4") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:22.808275 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.807857    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:31:21 +0000 UTC" deadline="2028-08-18 23:34:15.989627978 +0000 UTC"
Nov 03 19:36:22.808275 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.808217    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15699h57m53.181431657s"
Nov 03 19:36:22.844732 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.844115    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:22.844732 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:22.844290    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfcnb" podUID="f951b540-0179-474c-8a2c-7ee804f9d630"
Nov 03 19:36:22.867049 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.866937    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x88xn" event={"ID":"65b590ed-e1a5-467f-9283-7c875dd9ee24","Type":"ContainerStarted","Data":"dc002ae6adfc9db21f86304ef7cef337adb0de17d39be256b6634f686cf10d88"}
Nov 03 19:36:22.872372 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.872301    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vh26d" event={"ID":"b1cda986-f386-462d-9956-acc2a4de6e32","Type":"ContainerStarted","Data":"5a7f41c5228d6776608f44fd25798b14838c46bc7bdb1a5f798a7071d15e8c35"}
Nov 03 19:36:22.874366 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.874337    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xtsbd" event={"ID":"32452ae6-08d2-4fb2-863d-72dc09d4c0a2","Type":"ContainerStarted","Data":"da480f56accb6891c3c0d00769c98771f2bd08c62e5ec777516d31555cb5a27d"}
Nov 03 19:36:22.877890 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.877862    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4tbd" event={"ID":"b4a42db6-0377-4a5a-94ab-be3f82b332aa","Type":"ContainerStarted","Data":"8edd2f162297442f9f77e5ba98660f4c491934551e31553c054c9eecdd98d5b1"}
Nov 03 19:36:22.882568 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.882492    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pswxd" event={"ID":"88a27c5e-de31-4090-be4d-6b8d6551e4c7","Type":"ContainerStarted","Data":"e4d01dcd5adba2d4060227e841b164fc8265e16eec03d9d866e829e9e8e8af84"}
Nov 03 19:36:22.890076 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.890046    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rzn4t" event={"ID":"630922bd-4efc-4b44-96d1-fa55bce60e21","Type":"ContainerStarted","Data":"f11f7ea9334844e8aadb4a2b85717570432be27063013dd93e6846a27bc4bf14"}
Nov 03 19:36:22.895637 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.895607    2566 generic.go:356] "Generic (PLEG): container finished" podID="6c67d5fc48584d861d9d388770293f30" containerID="3c4f638977774a2da8ebe5666d3201fa97be208b0bd50951b9ab288d2f28ffad" exitCode=0
Nov 03 19:36:22.896505 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.896481    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal" event={"ID":"6c67d5fc48584d861d9d388770293f30","Type":"ContainerDied","Data":"3c4f638977774a2da8ebe5666d3201fa97be208b0bd50951b9ab288d2f28ffad"}
Nov 03 19:36:22.901575 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.901337    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal" event={"ID":"3def2722c7c58bcddedb41ddbc48c3ab","Type":"ContainerStarted","Data":"9eeb4aee2f3f207aedbf2c59014cc70609199c30fc6ecf19d39fc0f4a46c0b5f"}
Nov 03 19:36:22.904079 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.904050    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mgrnm" event={"ID":"71c679be-e63f-4ad4-a1b2-5474ec4fa125","Type":"ContainerStarted","Data":"dcbdef7fc62a5ddb080912db6bee1f7c41b56171960aabe80dc6ab7257fc9264"}
Nov 03 19:36:22.908250 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.908228    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-htzs4" event={"ID":"055215b4-9de9-494c-bbff-288cc97baf2f","Type":"ContainerStarted","Data":"f45ff49a9cf6737e1a1324c65dd202171c2106d3b3d10b63706762b06b9b7cf4"}
Nov 03 19:36:22.912325 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:22.912247    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf" event={"ID":"facedcd2-6b70-4a3b-980c-04dceba2ef0d","Type":"ContainerStarted","Data":"297782fc2eb6ce8a7dd368a98ac3593c216345a3f5ff82942c8d18edba787c48"}
Nov 03 19:36:23.393133 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:23.393095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca39ebc9-da71-43c2-893f-c033684c9399-metrics-certs\") pod \"network-metrics-daemon-794nj\" (UID: \"ca39ebc9-da71-43c2-893f-c033684c9399\") " pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:23.393344 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:23.393325    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:23.393416 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:23.393395    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca39ebc9-da71-43c2-893f-c033684c9399-metrics-certs podName:ca39ebc9-da71-43c2-893f-c033684c9399 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:25.393377146 +0000 UTC m=+6.038567444 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca39ebc9-da71-43c2-893f-c033684c9399-metrics-certs") pod "network-metrics-daemon-794nj" (UID: "ca39ebc9-da71-43c2-893f-c033684c9399") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:23.494441 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:23.494400    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng4gr\" (UniqueName: \"kubernetes.io/projected/f951b540-0179-474c-8a2c-7ee804f9d630-kube-api-access-ng4gr\") pod \"network-check-target-pfcnb\" (UID: \"f951b540-0179-474c-8a2c-7ee804f9d630\") " pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:23.494609 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:23.494473    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98d32fbe-5011-47a8-b937-725a8cad95a4-original-pull-secret\") pod \"global-pull-secret-syncer-6q5vq\" (UID: \"98d32fbe-5011-47a8-b937-725a8cad95a4\") " pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:23.494685 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:23.494623    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:23.494734 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:23.494690    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d32fbe-5011-47a8-b937-725a8cad95a4-original-pull-secret podName:98d32fbe-5011-47a8-b937-725a8cad95a4 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:25.494672575 +0000 UTC m=+6.139862881 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/98d32fbe-5011-47a8-b937-725a8cad95a4-original-pull-secret") pod "global-pull-secret-syncer-6q5vq" (UID: "98d32fbe-5011-47a8-b937-725a8cad95a4") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:23.495046 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:23.495027    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:36:23.495133 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:23.495053    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:36:23.495133 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:23.495065    2566 projected.go:196] Error preparing data for projected volume kube-api-access-ng4gr for pod openshift-network-diagnostics/network-check-target-pfcnb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:23.495133 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:23.495110    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f951b540-0179-474c-8a2c-7ee804f9d630-kube-api-access-ng4gr podName:f951b540-0179-474c-8a2c-7ee804f9d630 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:25.495095196 +0000 UTC m=+6.140285486 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ng4gr" (UniqueName: "kubernetes.io/projected/f951b540-0179-474c-8a2c-7ee804f9d630-kube-api-access-ng4gr") pod "network-check-target-pfcnb" (UID: "f951b540-0179-474c-8a2c-7ee804f9d630") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:23.773414 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:23.773380    2566 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:36:23.809237 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:23.809159    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:31:21 +0000 UTC" deadline="2028-03-30 21:43:17.359217739 +0000 UTC"
Nov 03 19:36:23.809237 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:23.809237    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12314h6m53.54998627s"
Nov 03 19:36:23.846178 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:23.846146    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:23.846372 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:23.846298    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6q5vq" podUID="98d32fbe-5011-47a8-b937-725a8cad95a4"
Nov 03 19:36:23.846722 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:23.846696    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:23.846828 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:23.846812    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-794nj" podUID="ca39ebc9-da71-43c2-893f-c033684c9399"
Nov 03 19:36:23.921613 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:23.921580    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal" event={"ID":"6c67d5fc48584d861d9d388770293f30","Type":"ContainerStarted","Data":"073845a477185d2c0b768e85e292b3b77ab77d86cdfd8712a65681da7ec577c0"}
Nov 03 19:36:23.940895 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:23.939740    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal" podStartSLOduration=2.939723775 podStartE2EDuration="2.939723775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:22.931373842 +0000 UTC m=+3.576564146" watchObservedRunningTime="2026-11-03 19:36:23.939723775 +0000 UTC m=+4.584914081"
Nov 03 19:36:23.940895 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:23.940473    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal" podStartSLOduration=2.940458 podStartE2EDuration="2.940458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:23.939400085 +0000 UTC m=+4.584590389" watchObservedRunningTime="2026-11-03 19:36:23.940458 +0000 UTC m=+4.585648305"
Nov 03 19:36:24.844479 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:24.843918    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:24.844479 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:24.844034    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfcnb" podUID="f951b540-0179-474c-8a2c-7ee804f9d630"
Nov 03 19:36:25.407959 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:25.407867    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca39ebc9-da71-43c2-893f-c033684c9399-metrics-certs\") pod \"network-metrics-daemon-794nj\" (UID: \"ca39ebc9-da71-43c2-893f-c033684c9399\") " pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:25.408127 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:25.408066    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:25.408190 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:25.408136    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca39ebc9-da71-43c2-893f-c033684c9399-metrics-certs podName:ca39ebc9-da71-43c2-893f-c033684c9399 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:29.408115759 +0000 UTC m=+10.053306057 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca39ebc9-da71-43c2-893f-c033684c9399-metrics-certs") pod "network-metrics-daemon-794nj" (UID: "ca39ebc9-da71-43c2-893f-c033684c9399") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:25.508469 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:25.508430    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng4gr\" (UniqueName: \"kubernetes.io/projected/f951b540-0179-474c-8a2c-7ee804f9d630-kube-api-access-ng4gr\") pod \"network-check-target-pfcnb\" (UID: \"f951b540-0179-474c-8a2c-7ee804f9d630\") " pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:25.508651 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:25.508501    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98d32fbe-5011-47a8-b937-725a8cad95a4-original-pull-secret\") pod \"global-pull-secret-syncer-6q5vq\" (UID: \"98d32fbe-5011-47a8-b937-725a8cad95a4\") " pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:25.508651 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:25.508640    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:25.508757 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:25.508694    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d32fbe-5011-47a8-b937-725a8cad95a4-original-pull-secret podName:98d32fbe-5011-47a8-b937-725a8cad95a4 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:29.508682392 +0000 UTC m=+10.153872677 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/98d32fbe-5011-47a8-b937-725a8cad95a4-original-pull-secret") pod "global-pull-secret-syncer-6q5vq" (UID: "98d32fbe-5011-47a8-b937-725a8cad95a4") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:25.509069 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:25.509045    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:36:25.509166 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:25.509073    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:36:25.509166 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:25.509082    2566 projected.go:196] Error preparing data for projected volume kube-api-access-ng4gr for pod openshift-network-diagnostics/network-check-target-pfcnb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:25.509166 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:25.509113    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f951b540-0179-474c-8a2c-7ee804f9d630-kube-api-access-ng4gr podName:f951b540-0179-474c-8a2c-7ee804f9d630 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:29.509103164 +0000 UTC m=+10.154293444 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ng4gr" (UniqueName: "kubernetes.io/projected/f951b540-0179-474c-8a2c-7ee804f9d630-kube-api-access-ng4gr") pod "network-check-target-pfcnb" (UID: "f951b540-0179-474c-8a2c-7ee804f9d630") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:25.846471 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:25.845834    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:25.846471 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:25.845973    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-794nj" podUID="ca39ebc9-da71-43c2-893f-c033684c9399"
Nov 03 19:36:25.846471 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:25.846335    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:25.846471 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:25.846418    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6q5vq" podUID="98d32fbe-5011-47a8-b937-725a8cad95a4"
Nov 03 19:36:25.933872 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:25.932964    2566 generic.go:356] "Generic (PLEG): container finished" podID="b4a42db6-0377-4a5a-94ab-be3f82b332aa" containerID="16e756e1f4bba1d8fdbe23eb419c63372566a3b8e24f0c0bb25783565b787913" exitCode=0
Nov 03 19:36:25.933872 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:25.933023    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4tbd" event={"ID":"b4a42db6-0377-4a5a-94ab-be3f82b332aa","Type":"ContainerDied","Data":"16e756e1f4bba1d8fdbe23eb419c63372566a3b8e24f0c0bb25783565b787913"}
Nov 03 19:36:26.844234 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:26.843711    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:26.844234 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:26.843841    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfcnb" podUID="f951b540-0179-474c-8a2c-7ee804f9d630"
Nov 03 19:36:26.946282 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:26.946245    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vh26d" event={"ID":"b1cda986-f386-462d-9956-acc2a4de6e32","Type":"ContainerStarted","Data":"df76ac550c523e78762008ed216c43c2be21ae53aedf62320cf9159db84881c8"}
Nov 03 19:36:27.118373 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:27.118275    2566 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vh26d"
Nov 03 19:36:27.119089 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:27.119061    2566 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vh26d"
Nov 03 19:36:27.138117 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:27.137330    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vh26d" podStartSLOduration=3.470057031 podStartE2EDuration="7.137310166s" totalImagesPullingTime="3.667253135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:26.966583816 +0000 UTC m=+7.611774120" watchObservedRunningTime="2026-11-03 19:36:27.137310166 +0000 UTC m=+7.782500472"
Nov 03 19:36:27.844338 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:27.844311    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:27.844457 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:27.844434    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6q5vq" podUID="98d32fbe-5011-47a8-b937-725a8cad95a4"
Nov 03 19:36:27.845115 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:27.845087    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:27.845248 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:27.845223    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-794nj" podUID="ca39ebc9-da71-43c2-893f-c033684c9399"
Nov 03 19:36:27.954143 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:27.954055    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf" event={"ID":"facedcd2-6b70-4a3b-980c-04dceba2ef0d","Type":"ContainerStarted","Data":"d1309566cc82b5f8a232578dd2f522673d36e8980047a6c5d899fce6e1c7882e"}
Nov 03 19:36:27.957375 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:27.957263    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pswxd" event={"ID":"88a27c5e-de31-4090-be4d-6b8d6551e4c7","Type":"ContainerStarted","Data":"c4612aed4c36e9e30975ffe72a9d16c3bbc4beea05a8d4d2a834654803acd512"}
Nov 03 19:36:27.975416 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:27.975364    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pswxd" podStartSLOduration=2.873426659 podStartE2EDuration="7.975345924s" totalImagesPullingTime="5.101919265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:27.97429056 +0000 UTC m=+8.619480864" watchObservedRunningTime="2026-11-03 19:36:27.975345924 +0000 UTC m=+8.620536230"
Nov 03 19:36:28.844004 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:28.843547    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:28.844004 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:28.843663    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfcnb" podUID="f951b540-0179-474c-8a2c-7ee804f9d630"
Nov 03 19:36:28.958857 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:28.958567    2566 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 19:36:29.201264 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:29.201226    2566 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 19:36:29.443505 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:29.443475    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca39ebc9-da71-43c2-893f-c033684c9399-metrics-certs\") pod \"network-metrics-daemon-794nj\" (UID: \"ca39ebc9-da71-43c2-893f-c033684c9399\") " pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:29.443655 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:29.443636    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:29.443728 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:29.443709    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca39ebc9-da71-43c2-893f-c033684c9399-metrics-certs podName:ca39ebc9-da71-43c2-893f-c033684c9399 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:37.443687994 +0000 UTC m=+18.088878275 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca39ebc9-da71-43c2-893f-c033684c9399-metrics-certs") pod "network-metrics-daemon-794nj" (UID: "ca39ebc9-da71-43c2-893f-c033684c9399") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:29.544745 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:29.544570    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng4gr\" (UniqueName: \"kubernetes.io/projected/f951b540-0179-474c-8a2c-7ee804f9d630-kube-api-access-ng4gr\") pod \"network-check-target-pfcnb\" (UID: \"f951b540-0179-474c-8a2c-7ee804f9d630\") " pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:29.544745 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:29.544645    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98d32fbe-5011-47a8-b937-725a8cad95a4-original-pull-secret\") pod \"global-pull-secret-syncer-6q5vq\" (UID: \"98d32fbe-5011-47a8-b937-725a8cad95a4\") " pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:29.544745 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:29.544750    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:36:29.545023 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:29.544771    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:36:29.545023 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:29.544783    2566 projected.go:196] Error preparing data for projected volume kube-api-access-ng4gr for pod openshift-network-diagnostics/network-check-target-pfcnb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:29.545023 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:29.544819    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:29.545023 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:29.544849    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f951b540-0179-474c-8a2c-7ee804f9d630-kube-api-access-ng4gr podName:f951b540-0179-474c-8a2c-7ee804f9d630 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:37.544833361 +0000 UTC m=+18.190023655 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ng4gr" (UniqueName: "kubernetes.io/projected/f951b540-0179-474c-8a2c-7ee804f9d630-kube-api-access-ng4gr") pod "network-check-target-pfcnb" (UID: "f951b540-0179-474c-8a2c-7ee804f9d630") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:29.554140 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:29.545302    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d32fbe-5011-47a8-b937-725a8cad95a4-original-pull-secret podName:98d32fbe-5011-47a8-b937-725a8cad95a4 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:37.54486732 +0000 UTC m=+18.190057621 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/98d32fbe-5011-47a8-b937-725a8cad95a4-original-pull-secret") pod "global-pull-secret-syncer-6q5vq" (UID: "98d32fbe-5011-47a8-b937-725a8cad95a4") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:29.847795 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:29.847702    2566 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:36:29.201246671Z","UUID":"a8a80a9c-f3db-49c0-978b-70659a6e7a42","Handler":null,"Name":"","Endpoint":""}
Nov 03 19:36:29.852545 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:29.850359    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 19:36:29.852545 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:29.850390    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 19:36:29.852545 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:29.851401    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:29.852545 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:29.851521    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6q5vq" podUID="98d32fbe-5011-47a8-b937-725a8cad95a4"
Nov 03 19:36:29.852545 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:29.851667    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:29.852545 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:29.851831    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-794nj" podUID="ca39ebc9-da71-43c2-893f-c033684c9399"
Nov 03 19:36:29.963918 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:29.963878    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf" event={"ID":"facedcd2-6b70-4a3b-980c-04dceba2ef0d","Type":"ContainerStarted","Data":"7ceb8e113fead2e0bdbefbaf44058667bfdee50ddc8445ac0cfbdbdcdfb25207"}
Nov 03 19:36:29.965794 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:29.965765    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xtsbd" event={"ID":"32452ae6-08d2-4fb2-863d-72dc09d4c0a2","Type":"ContainerStarted","Data":"9bb9a91477278636d01573a2e193ce018d63999cfe0ceaa50f2c2469cab4bebb"}
Nov 03 19:36:30.844085 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:30.844060    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:30.844237 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:30.844163    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfcnb" podUID="f951b540-0179-474c-8a2c-7ee804f9d630"
Nov 03 19:36:30.972804 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:30.972761    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rzn4t" event={"ID":"630922bd-4efc-4b44-96d1-fa55bce60e21","Type":"ContainerStarted","Data":"39525bcedf37c2b4a5878edfc5ab812e25dd818b3c1def9c2ba00df8aa3b0347"}
Nov 03 19:36:30.975977 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:30.975467    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf" event={"ID":"facedcd2-6b70-4a3b-980c-04dceba2ef0d","Type":"ContainerStarted","Data":"6ddc3bd0baec2f295bf4cf10d41c234dd82c8f3a59082563384d32de55a2b80d"}
Nov 03 19:36:30.977944 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:30.977522    2566 generic.go:356] "Generic (PLEG): container finished" podID="b4a42db6-0377-4a5a-94ab-be3f82b332aa" containerID="3afd2dbeb9ef4b7316a2063dcf6d99ccc5b44059cdbabae60f2a058477f1e803" exitCode=0
Nov 03 19:36:30.977944 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:30.977608    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4tbd" event={"ID":"b4a42db6-0377-4a5a-94ab-be3f82b332aa","Type":"ContainerDied","Data":"3afd2dbeb9ef4b7316a2063dcf6d99ccc5b44059cdbabae60f2a058477f1e803"}
Nov 03 19:36:30.989830 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:30.989786    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-rzn4t" podStartSLOduration=4.220767961 podStartE2EDuration="11.989768856s" totalImagesPullingTime="7.769000895s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:30.98949703 +0000 UTC m=+11.634687333" watchObservedRunningTime="2026-11-03 19:36:30.989768856 +0000 UTC m=+11.634959162"
Nov 03 19:36:30.990022 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:30.990000    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-xtsbd" podStartSLOduration=4.643942521 podStartE2EDuration="11.9899936s" totalImagesPullingTime="7.346051079s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:29.98760929 +0000 UTC m=+10.632799594" watchObservedRunningTime="2026-11-03 19:36:30.9899936 +0000 UTC m=+11.635183902"
Nov 03 19:36:31.010419 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:31.010365    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf" podStartSLOduration=3.120853226 podStartE2EDuration="11.010344925s" totalImagesPullingTime="7.889491699s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:31.009951264 +0000 UTC m=+11.655141569" watchObservedRunningTime="2026-11-03 19:36:31.010344925 +0000 UTC m=+11.655535232"
Nov 03 19:36:31.843873 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:31.843644    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:31.844047 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:31.843966    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6q5vq" podUID="98d32fbe-5011-47a8-b937-725a8cad95a4"
Nov 03 19:36:31.844247 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:31.843678    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:31.844434 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:31.844363    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-794nj" podUID="ca39ebc9-da71-43c2-893f-c033684c9399"
Nov 03 19:36:31.980828 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:31.980792    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-htzs4" event={"ID":"055215b4-9de9-494c-bbff-288cc97baf2f","Type":"ContainerStarted","Data":"86881e5988a14415ef9861314d2c2b4b5fc6126ca9bc688f4815c900b2fe963b"}
Nov 03 19:36:31.997954 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:31.997904    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-htzs4" podStartSLOduration=3.900981417 podStartE2EDuration="11.99788693s" totalImagesPullingTime="8.096905513s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:31.997885669 +0000 UTC m=+12.643075972" watchObservedRunningTime="2026-11-03 19:36:31.99788693 +0000 UTC m=+12.643077240"
Nov 03 19:36:32.844502 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:32.844465    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:32.844704 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:32.844592    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfcnb" podUID="f951b540-0179-474c-8a2c-7ee804f9d630"
Nov 03 19:36:32.985012 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:32.984978    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mgrnm" event={"ID":"71c679be-e63f-4ad4-a1b2-5474ec4fa125","Type":"ContainerStarted","Data":"576aaf265ca5502300e41e8d7493b392c2a909cf873bfa243003c7d350340495"}
Nov 03 19:36:32.987808 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:32.987784    2566 generic.go:356] "Generic (PLEG): container finished" podID="b4a42db6-0377-4a5a-94ab-be3f82b332aa" containerID="cf8c2a98f6b4f3d42b63ea1fdf8902c3366e20f3052ea52259a3e6ba02564b7c" exitCode=0
Nov 03 19:36:32.987932 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:32.987873    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4tbd" event={"ID":"b4a42db6-0377-4a5a-94ab-be3f82b332aa","Type":"ContainerDied","Data":"cf8c2a98f6b4f3d42b63ea1fdf8902c3366e20f3052ea52259a3e6ba02564b7c"}
Nov 03 19:36:33.028856 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:33.028799    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mgrnm" podStartSLOduration=2.959240577 podStartE2EDuration="13.028786671s" totalImagesPullingTime="10.069546094s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:33.004408872 +0000 UTC m=+13.649599179" watchObservedRunningTime="2026-11-03 19:36:33.028786671 +0000 UTC m=+13.673976977"
Nov 03 19:36:33.844623 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:33.844591    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:33.844787 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:33.844730    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-794nj" podUID="ca39ebc9-da71-43c2-893f-c033684c9399"
Nov 03 19:36:33.845046 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:33.845028    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:33.845149 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:33.845128    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6q5vq" podUID="98d32fbe-5011-47a8-b937-725a8cad95a4"
Nov 03 19:36:33.994837 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:33.993632    2566 generic.go:356] "Generic (PLEG): container finished" podID="b4a42db6-0377-4a5a-94ab-be3f82b332aa" containerID="dca166d961a8c0bf8f4fce68c01ef8c985a4de0c7ca472a7e7d40883aca28a47" exitCode=0
Nov 03 19:36:33.994837 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:33.994609    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4tbd" event={"ID":"b4a42db6-0377-4a5a-94ab-be3f82b332aa","Type":"ContainerDied","Data":"dca166d961a8c0bf8f4fce68c01ef8c985a4de0c7ca472a7e7d40883aca28a47"}
Nov 03 19:36:34.844062 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:34.844032    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:34.844259 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:34.844138    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfcnb" podUID="f951b540-0179-474c-8a2c-7ee804f9d630"
Nov 03 19:36:35.844393 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:35.844360    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:35.844393 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:35.844390    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:35.844961 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:35.844483    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6q5vq" podUID="98d32fbe-5011-47a8-b937-725a8cad95a4"
Nov 03 19:36:35.844961 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:35.844602    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-794nj" podUID="ca39ebc9-da71-43c2-893f-c033684c9399"
Nov 03 19:36:36.001695 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:36.001653    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4tbd" event={"ID":"b4a42db6-0377-4a5a-94ab-be3f82b332aa","Type":"ContainerStarted","Data":"054a11272a751929b7e38eb5850975116b7823e823ce7d82266f17029e1771b8"}
Nov 03 19:36:36.578985 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:36.578818    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vh26d"
Nov 03 19:36:36.579139 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:36.579113    2566 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 19:36:36.579573 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:36.579529    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vh26d"
Nov 03 19:36:36.843716 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:36.843640    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:36.843846 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:36.843733    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfcnb" podUID="f951b540-0179-474c-8a2c-7ee804f9d630"
Nov 03 19:36:37.006145 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:37.006116    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 19:36:37.006639 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:37.006485    2566 generic.go:356] "Generic (PLEG): container finished" podID="65b590ed-e1a5-467f-9283-7c875dd9ee24" containerID="026bd52efdad675243c5ef5fc94f526681bad64e10431d752cd39c325214431a" exitCode=1
Nov 03 19:36:37.006639 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:37.006553    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x88xn" event={"ID":"65b590ed-e1a5-467f-9283-7c875dd9ee24","Type":"ContainerStarted","Data":"03909f2c5c5542bc052c57bccf52620ed12a0c39c535230ea9135c15d2cb1dc5"}
Nov 03 19:36:37.006639 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:37.006629    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x88xn" event={"ID":"65b590ed-e1a5-467f-9283-7c875dd9ee24","Type":"ContainerStarted","Data":"85baa5845e0b50c630e071bb49b37cf993c3e55faa4695d1c170beba14d71833"}
Nov 03 19:36:37.006768 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:37.006666    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x88xn" event={"ID":"65b590ed-e1a5-467f-9283-7c875dd9ee24","Type":"ContainerStarted","Data":"e5ac14139424f5f3a3badadabe12a8a547fa9c4a872bceb417dde14fa92958c3"}
Nov 03 19:36:37.006768 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:37.006694    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x88xn" event={"ID":"65b590ed-e1a5-467f-9283-7c875dd9ee24","Type":"ContainerStarted","Data":"046ee159ee9810b8d353f52c9d74826f265fb2865d2621d526c3455349df62fe"}
Nov 03 19:36:37.006768 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:37.006727    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x88xn" event={"ID":"65b590ed-e1a5-467f-9283-7c875dd9ee24","Type":"ContainerDied","Data":"026bd52efdad675243c5ef5fc94f526681bad64e10431d752cd39c325214431a"}
Nov 03 19:36:37.006871 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:37.006772    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x88xn" event={"ID":"65b590ed-e1a5-467f-9283-7c875dd9ee24","Type":"ContainerStarted","Data":"ea2e7db2963d289d38b6a1dd5f37281fc9c51d677964dfee6092090f453f6968"}
Nov 03 19:36:37.009219 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:37.009182    2566 generic.go:356] "Generic (PLEG): container finished" podID="b4a42db6-0377-4a5a-94ab-be3f82b332aa" containerID="054a11272a751929b7e38eb5850975116b7823e823ce7d82266f17029e1771b8" exitCode=0
Nov 03 19:36:37.009328 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:37.009242    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4tbd" event={"ID":"b4a42db6-0377-4a5a-94ab-be3f82b332aa","Type":"ContainerDied","Data":"054a11272a751929b7e38eb5850975116b7823e823ce7d82266f17029e1771b8"}
Nov 03 19:36:37.508445 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:37.508232    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca39ebc9-da71-43c2-893f-c033684c9399-metrics-certs\") pod \"network-metrics-daemon-794nj\" (UID: \"ca39ebc9-da71-43c2-893f-c033684c9399\") " pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:37.508445 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:37.508373    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:37.508679 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:37.508501    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca39ebc9-da71-43c2-893f-c033684c9399-metrics-certs podName:ca39ebc9-da71-43c2-893f-c033684c9399 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:53.508484037 +0000 UTC m=+34.153674331 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca39ebc9-da71-43c2-893f-c033684c9399-metrics-certs") pod "network-metrics-daemon-794nj" (UID: "ca39ebc9-da71-43c2-893f-c033684c9399") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:37.609453 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:37.609415    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng4gr\" (UniqueName: \"kubernetes.io/projected/f951b540-0179-474c-8a2c-7ee804f9d630-kube-api-access-ng4gr\") pod \"network-check-target-pfcnb\" (UID: \"f951b540-0179-474c-8a2c-7ee804f9d630\") " pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:37.609583 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:37.609468    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98d32fbe-5011-47a8-b937-725a8cad95a4-original-pull-secret\") pod \"global-pull-secret-syncer-6q5vq\" (UID: \"98d32fbe-5011-47a8-b937-725a8cad95a4\") " pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:37.609583 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:37.609554    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:37.609583 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:37.609573    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:36:37.609687 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:37.609597    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:36:37.609687 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:37.609607    2566 projected.go:196] Error preparing data for projected volume kube-api-access-ng4gr for pod openshift-network-diagnostics/network-check-target-pfcnb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:37.609687 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:37.609610    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d32fbe-5011-47a8-b937-725a8cad95a4-original-pull-secret podName:98d32fbe-5011-47a8-b937-725a8cad95a4 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:53.609592624 +0000 UTC m=+34.254782905 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/98d32fbe-5011-47a8-b937-725a8cad95a4-original-pull-secret") pod "global-pull-secret-syncer-6q5vq" (UID: "98d32fbe-5011-47a8-b937-725a8cad95a4") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:37.609687 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:37.609646    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f951b540-0179-474c-8a2c-7ee804f9d630-kube-api-access-ng4gr podName:f951b540-0179-474c-8a2c-7ee804f9d630 nodeName:}" failed. No retries permitted until 2026-11-03 19:36:53.60963359 +0000 UTC m=+34.254823871 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ng4gr" (UniqueName: "kubernetes.io/projected/f951b540-0179-474c-8a2c-7ee804f9d630-kube-api-access-ng4gr") pod "network-check-target-pfcnb" (UID: "f951b540-0179-474c-8a2c-7ee804f9d630") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:37.844007 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:37.843978    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:37.844179 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:37.844079    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-794nj" podUID="ca39ebc9-da71-43c2-893f-c033684c9399"
Nov 03 19:36:37.844266 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:37.844187    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:37.844339 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:37.844318    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6q5vq" podUID="98d32fbe-5011-47a8-b937-725a8cad95a4"
Nov 03 19:36:38.014737 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:38.014708    2566 generic.go:356] "Generic (PLEG): container finished" podID="b4a42db6-0377-4a5a-94ab-be3f82b332aa" containerID="64ec19ed181218aa44a8a488bce53a7fc0bc4ae31c1a6e1fbfbe783677805919" exitCode=0
Nov 03 19:36:38.015135 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:38.014747    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4tbd" event={"ID":"b4a42db6-0377-4a5a-94ab-be3f82b332aa","Type":"ContainerDied","Data":"64ec19ed181218aa44a8a488bce53a7fc0bc4ae31c1a6e1fbfbe783677805919"}
Nov 03 19:36:38.843634 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:38.843605    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:38.843761 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:38.843706    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfcnb" podUID="f951b540-0179-474c-8a2c-7ee804f9d630"
Nov 03 19:36:39.019654 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:39.019630    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 19:36:39.020145 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:39.020015    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x88xn" event={"ID":"65b590ed-e1a5-467f-9283-7c875dd9ee24","Type":"ContainerStarted","Data":"a041c6878a2d7858bd224b3abf9ea02cf6e70690962dc72e1a23a0bf79f7c6da"}
Nov 03 19:36:39.023050 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:39.023028    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q4tbd" event={"ID":"b4a42db6-0377-4a5a-94ab-be3f82b332aa","Type":"ContainerStarted","Data":"23fde1d367fb2a4bdbd25308da5a2eafee5e2ff676bf050b12f62292924324a8"}
Nov 03 19:36:39.051875 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:39.051835    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-q4tbd" podStartSLOduration=8.118348983 podStartE2EDuration="19.051823931s" totalImagesPullingTime="10.933474948s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:39.051496186 +0000 UTC m=+19.696686489" watchObservedRunningTime="2026-11-03 19:36:39.051823931 +0000 UTC m=+19.697014234"
Nov 03 19:36:39.844351 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:39.844321    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:39.844533 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:39.844430    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-794nj" podUID="ca39ebc9-da71-43c2-893f-c033684c9399"
Nov 03 19:36:39.844533 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:39.844481    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:39.844621 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:39.844581    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6q5vq" podUID="98d32fbe-5011-47a8-b937-725a8cad95a4"
Nov 03 19:36:40.844109 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:40.844087    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:40.844632 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:40.844179    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfcnb" podUID="f951b540-0179-474c-8a2c-7ee804f9d630"
Nov 03 19:36:41.030105 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:41.029934    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 19:36:41.030448 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:41.030422    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x88xn" event={"ID":"65b590ed-e1a5-467f-9283-7c875dd9ee24","Type":"ContainerStarted","Data":"df60cf05a8f94dd169aa58eedf0f0935b7e2a6d62cd42acc79532796226c0f48"}
Nov 03 19:36:41.030706 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:41.030686    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:41.030818 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:41.030804    2566 scope.go:122] "RemoveContainer" containerID="026bd52efdad675243c5ef5fc94f526681bad64e10431d752cd39c325214431a"
Nov 03 19:36:41.048338 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:41.048319    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:41.844194 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:41.844166    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:41.844562 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:41.844195    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:41.844562 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:41.844293    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-794nj" podUID="ca39ebc9-da71-43c2-893f-c033684c9399"
Nov 03 19:36:41.844562 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:41.844414    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6q5vq" podUID="98d32fbe-5011-47a8-b937-725a8cad95a4"
Nov 03 19:36:42.043439 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:42.043403    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 19:36:42.043787 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:42.043761    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x88xn" event={"ID":"65b590ed-e1a5-467f-9283-7c875dd9ee24","Type":"ContainerStarted","Data":"1d14c7c650f25b9dfabf6f06b0e8965a2b4eee7e294fe136f700f479240bd864"}
Nov 03 19:36:42.044035 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:42.044014    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:42.044128 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:42.044096    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:42.060656 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:42.060625    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:36:42.085236 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:42.084398    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x88xn" podStartSLOduration=8.880088826 podStartE2EDuration="23.084376285s" totalImagesPullingTime="14.204287459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:36:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:36:42.083873431 +0000 UTC m=+22.729063735" watchObservedRunningTime="2026-11-03 19:36:42.084376285 +0000 UTC m=+22.729566588"
Nov 03 19:36:42.595989 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:42.595906    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6q5vq"]
Nov 03 19:36:42.596248 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:42.596077    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:42.596248 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:42.596191    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6q5vq" podUID="98d32fbe-5011-47a8-b937-725a8cad95a4"
Nov 03 19:36:42.596565 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:42.596538    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pfcnb"]
Nov 03 19:36:42.596665 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:42.596643    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:42.596744 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:42.596727    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfcnb" podUID="f951b540-0179-474c-8a2c-7ee804f9d630"
Nov 03 19:36:42.598808 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:42.598788    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-794nj"]
Nov 03 19:36:42.598903 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:42.598891    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:42.599014 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:42.598997    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-794nj" podUID="ca39ebc9-da71-43c2-893f-c033684c9399"
Nov 03 19:36:43.636240 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:43.636187    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rzn4t_630922bd-4efc-4b44-96d1-fa55bce60e21/dns-node-resolver/0.log"
Nov 03 19:36:43.844458 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:43.844427    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:43.844595 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:43.844556    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-794nj" podUID="ca39ebc9-da71-43c2-893f-c033684c9399"
Nov 03 19:36:44.607989 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:44.607962    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pswxd_88a27c5e-de31-4090-be4d-6b8d6551e4c7/node-ca/0.log"
Nov 03 19:36:44.843643 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:44.843607    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:44.843643 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:44.843640    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:44.844175 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:44.843727    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6q5vq" podUID="98d32fbe-5011-47a8-b937-725a8cad95a4"
Nov 03 19:36:44.844175 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:44.843829    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfcnb" podUID="f951b540-0179-474c-8a2c-7ee804f9d630"
Nov 03 19:36:45.844277 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:45.844242    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:45.844740 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:45.844356    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-794nj" podUID="ca39ebc9-da71-43c2-893f-c033684c9399"
Nov 03 19:36:46.844295 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:46.844264    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:46.844295 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:46.844280    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:46.844709 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:46.844362    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6q5vq" podUID="98d32fbe-5011-47a8-b937-725a8cad95a4"
Nov 03 19:36:46.844709 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:46.844473    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfcnb" podUID="f951b540-0179-474c-8a2c-7ee804f9d630"
Nov 03 19:36:47.843819 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:47.843783    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:47.844067 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:47.843915    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-794nj" podUID="ca39ebc9-da71-43c2-893f-c033684c9399"
Nov 03 19:36:48.844264 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:48.844222    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:48.844759 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:48.844231    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:48.844759 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:48.844352    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6q5vq" podUID="98d32fbe-5011-47a8-b937-725a8cad95a4"
Nov 03 19:36:48.844759 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:48.844416    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfcnb" podUID="f951b540-0179-474c-8a2c-7ee804f9d630"
Nov 03 19:36:49.844531 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:49.844501    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:49.844970 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:49.844596    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-794nj" podUID="ca39ebc9-da71-43c2-893f-c033684c9399"
Nov 03 19:36:50.844326 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:50.844292    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:50.844500 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:50.844399    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfcnb" podUID="f951b540-0179-474c-8a2c-7ee804f9d630"
Nov 03 19:36:50.844500 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:50.844426    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:50.844500 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:50.844490    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6q5vq" podUID="98d32fbe-5011-47a8-b937-725a8cad95a4"
Nov 03 19:36:51.844305 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:51.844263    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:51.844473 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:51.844381    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-794nj" podUID="ca39ebc9-da71-43c2-893f-c033684c9399"
Nov 03 19:36:52.844418 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:52.844378    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:52.844795 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:52.844481    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6q5vq" podUID="98d32fbe-5011-47a8-b937-725a8cad95a4"
Nov 03 19:36:52.844795 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:52.844533    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:52.844795 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:52.844617    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfcnb" podUID="f951b540-0179-474c-8a2c-7ee804f9d630"
Nov 03 19:36:53.515818 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:53.515779    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca39ebc9-da71-43c2-893f-c033684c9399-metrics-certs\") pod \"network-metrics-daemon-794nj\" (UID: \"ca39ebc9-da71-43c2-893f-c033684c9399\") " pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:53.515999 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:53.515878    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:53.515999 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:53.515927    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca39ebc9-da71-43c2-893f-c033684c9399-metrics-certs podName:ca39ebc9-da71-43c2-893f-c033684c9399 nodeName:}" failed. No retries permitted until 2026-11-03 19:37:25.515914875 +0000 UTC m=+66.161105156 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ca39ebc9-da71-43c2-893f-c033684c9399-metrics-certs") pod "network-metrics-daemon-794nj" (UID: "ca39ebc9-da71-43c2-893f-c033684c9399") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 19:36:53.616252 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:53.616198    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98d32fbe-5011-47a8-b937-725a8cad95a4-original-pull-secret\") pod \"global-pull-secret-syncer-6q5vq\" (UID: \"98d32fbe-5011-47a8-b937-725a8cad95a4\") " pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:53.616424 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:53.616295    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng4gr\" (UniqueName: \"kubernetes.io/projected/f951b540-0179-474c-8a2c-7ee804f9d630-kube-api-access-ng4gr\") pod \"network-check-target-pfcnb\" (UID: \"f951b540-0179-474c-8a2c-7ee804f9d630\") " pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:53.616424 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:53.616360    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:53.616424 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:53.616399    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 19:36:53.616424 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:53.616412    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 19:36:53.616424 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:53.616423    2566 projected.go:196] Error preparing data for projected volume kube-api-access-ng4gr for pod openshift-network-diagnostics/network-check-target-pfcnb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:53.616635 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:53.616424    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98d32fbe-5011-47a8-b937-725a8cad95a4-original-pull-secret podName:98d32fbe-5011-47a8-b937-725a8cad95a4 nodeName:}" failed. No retries permitted until 2026-11-03 19:37:25.616408058 +0000 UTC m=+66.261598359 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/98d32fbe-5011-47a8-b937-725a8cad95a4-original-pull-secret") pod "global-pull-secret-syncer-6q5vq" (UID: "98d32fbe-5011-47a8-b937-725a8cad95a4") : object "kube-system"/"original-pull-secret" not registered
Nov 03 19:36:53.616635 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:53.616463    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f951b540-0179-474c-8a2c-7ee804f9d630-kube-api-access-ng4gr podName:f951b540-0179-474c-8a2c-7ee804f9d630 nodeName:}" failed. No retries permitted until 2026-11-03 19:37:25.616453724 +0000 UTC m=+66.261644013 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ng4gr" (UniqueName: "kubernetes.io/projected/f951b540-0179-474c-8a2c-7ee804f9d630-kube-api-access-ng4gr") pod "network-check-target-pfcnb" (UID: "f951b540-0179-474c-8a2c-7ee804f9d630") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 19:36:53.846960 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:53.846879    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:53.847383 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:53.846997    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-794nj" podUID="ca39ebc9-da71-43c2-893f-c033684c9399"
Nov 03 19:36:54.843468 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:54.843439    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:54.843647 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:54.843439    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:54.843647 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:54.843544    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6q5vq" podUID="98d32fbe-5011-47a8-b937-725a8cad95a4"
Nov 03 19:36:54.843647 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:54.843603    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfcnb" podUID="f951b540-0179-474c-8a2c-7ee804f9d630"
Nov 03 19:36:55.843734 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:55.843699    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:55.844135 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:55.843818    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-794nj" podUID="ca39ebc9-da71-43c2-893f-c033684c9399"
Nov 03 19:36:56.844118 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:56.844091    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:56.844529 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:56.844091    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:56.844529 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:56.844191    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6q5vq" podUID="98d32fbe-5011-47a8-b937-725a8cad95a4"
Nov 03 19:36:56.844529 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:56.844269    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfcnb" podUID="f951b540-0179-474c-8a2c-7ee804f9d630"
Nov 03 19:36:57.844105 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:57.844074    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:57.844285 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:57.844187    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-794nj" podUID="ca39ebc9-da71-43c2-893f-c033684c9399"
Nov 03 19:36:58.843665 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:58.843624    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:36:58.843665 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:58.843660    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:36:58.843868 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:58.843732    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfcnb" podUID="f951b540-0179-474c-8a2c-7ee804f9d630"
Nov 03 19:36:58.843868 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:58.843799    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6q5vq" podUID="98d32fbe-5011-47a8-b937-725a8cad95a4"
Nov 03 19:36:59.844464 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:36:59.844433    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:36:59.844909 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:36:59.844542    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-794nj" podUID="ca39ebc9-da71-43c2-893f-c033684c9399"
Nov 03 19:37:00.844353 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:00.844318    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:37:00.844520 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:00.844320    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:37:00.844520 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:37:00.844421    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfcnb" podUID="f951b540-0179-474c-8a2c-7ee804f9d630"
Nov 03 19:37:00.844520 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:37:00.844497    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6q5vq" podUID="98d32fbe-5011-47a8-b937-725a8cad95a4"
Nov 03 19:37:01.843636 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:01.843596    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:37:01.843804 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:37:01.843740    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-794nj" podUID="ca39ebc9-da71-43c2-893f-c033684c9399"
Nov 03 19:37:02.844502 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:02.844468    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:37:02.844502 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:02.844500    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:37:02.844914 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:37:02.844581    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6q5vq" podUID="98d32fbe-5011-47a8-b937-725a8cad95a4"
Nov 03 19:37:02.844914 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:37:02.844699    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pfcnb" podUID="f951b540-0179-474c-8a2c-7ee804f9d630"
Nov 03 19:37:03.130612 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.130527    2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-206.us-east-2.compute.internal" event="NodeReady"
Nov 03 19:37:03.130773 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.130679    2566 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 19:37:03.176908 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.176882    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mksqw"]
Nov 03 19:37:03.178655 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.178640    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mksqw"
Nov 03 19:37:03.181831 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.181800    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2224c"]
Nov 03 19:37:03.183594 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.183567    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:03.184095 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.184060    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 19:37:03.184192 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.184158    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dgjzc\""
Nov 03 19:37:03.184311 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.184286    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 19:37:03.184385 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.184345    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 19:37:03.192642 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.192616    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mksqw"]
Nov 03 19:37:03.204374 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.204345    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2224c"]
Nov 03 19:37:03.204886 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.204865    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-868lx"]
Nov 03 19:37:03.206809 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.206791    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-868lx"
Nov 03 19:37:03.217021 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.216997    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-868lx"]
Nov 03 19:37:03.275374 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.275335    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4545\" (UniqueName: \"kubernetes.io/projected/f926eeff-6488-48c5-9e97-269a6d79dee1-kube-api-access-x4545\") pod \"ingress-canary-mksqw\" (UID: \"f926eeff-6488-48c5-9e97-269a6d79dee1\") " pod="openshift-ingress-canary/ingress-canary-mksqw"
Nov 03 19:37:03.275374 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.275373    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ac8ac4b-fef2-475d-a40b-9db77cded4d3-tmp-dir\") pod \"dns-default-868lx\" (UID: \"3ac8ac4b-fef2-475d-a40b-9db77cded4d3\") " pod="openshift-dns/dns-default-868lx"
Nov 03 19:37:03.275582 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.275398    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sgbt\" (UniqueName: \"kubernetes.io/projected/8771d72e-12a3-429f-b458-e22298238393-kube-api-access-5sgbt\") pod \"insights-runtime-extractor-2224c\" (UID: \"8771d72e-12a3-429f-b458-e22298238393\") " pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:03.275582 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.275486    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f926eeff-6488-48c5-9e97-269a6d79dee1-cert\") pod \"ingress-canary-mksqw\" (UID: \"f926eeff-6488-48c5-9e97-269a6d79dee1\") " pod="openshift-ingress-canary/ingress-canary-mksqw"
Nov 03 19:37:03.275582 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.275520    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp6g4\" (UniqueName: \"kubernetes.io/projected/3ac8ac4b-fef2-475d-a40b-9db77cded4d3-kube-api-access-jp6g4\") pod \"dns-default-868lx\" (UID: \"3ac8ac4b-fef2-475d-a40b-9db77cded4d3\") " pod="openshift-dns/dns-default-868lx"
Nov 03 19:37:03.275582 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.275538    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8771d72e-12a3-429f-b458-e22298238393-crio-socket\") pod \"insights-runtime-extractor-2224c\" (UID: \"8771d72e-12a3-429f-b458-e22298238393\") " pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:03.275694 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.275643    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ac8ac4b-fef2-475d-a40b-9db77cded4d3-metrics-tls\") pod \"dns-default-868lx\" (UID: \"3ac8ac4b-fef2-475d-a40b-9db77cded4d3\") " pod="openshift-dns/dns-default-868lx"
Nov 03 19:37:03.275694 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.275670    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8771d72e-12a3-429f-b458-e22298238393-data-volume\") pod \"insights-runtime-extractor-2224c\" (UID: \"8771d72e-12a3-429f-b458-e22298238393\") " pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:03.275751 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.275695    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8771d72e-12a3-429f-b458-e22298238393-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2224c\" (UID: \"8771d72e-12a3-429f-b458-e22298238393\") " pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:03.275751 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.275715    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8771d72e-12a3-429f-b458-e22298238393-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2224c\" (UID: \"8771d72e-12a3-429f-b458-e22298238393\") " pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:03.275751 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.275740    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac8ac4b-fef2-475d-a40b-9db77cded4d3-config-volume\") pod \"dns-default-868lx\" (UID: \"3ac8ac4b-fef2-475d-a40b-9db77cded4d3\") " pod="openshift-dns/dns-default-868lx"
Nov 03 19:37:03.284161 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.284131    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 19:37:03.284161 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.284148    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 19:37:03.284288 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.284140    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 19:37:03.284288 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.284140    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 19:37:03.284593 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.284565    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 19:37:03.287235 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.284733    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qg9jj\""
Nov 03 19:37:03.287235 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.284755    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b2f85\""
Nov 03 19:37:03.287235 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.284926    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 19:37:03.376683 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.376625    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4545\" (UniqueName: \"kubernetes.io/projected/f926eeff-6488-48c5-9e97-269a6d79dee1-kube-api-access-x4545\") pod \"ingress-canary-mksqw\" (UID: \"f926eeff-6488-48c5-9e97-269a6d79dee1\") " pod="openshift-ingress-canary/ingress-canary-mksqw"
Nov 03 19:37:03.376683 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.376683    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ac8ac4b-fef2-475d-a40b-9db77cded4d3-tmp-dir\") pod \"dns-default-868lx\" (UID: \"3ac8ac4b-fef2-475d-a40b-9db77cded4d3\") " pod="openshift-dns/dns-default-868lx"
Nov 03 19:37:03.377125 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.376709    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sgbt\" (UniqueName: \"kubernetes.io/projected/8771d72e-12a3-429f-b458-e22298238393-kube-api-access-5sgbt\") pod \"insights-runtime-extractor-2224c\" (UID: \"8771d72e-12a3-429f-b458-e22298238393\") " pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:03.377125 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.376845    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f926eeff-6488-48c5-9e97-269a6d79dee1-cert\") pod \"ingress-canary-mksqw\" (UID: \"f926eeff-6488-48c5-9e97-269a6d79dee1\") " pod="openshift-ingress-canary/ingress-canary-mksqw"
Nov 03 19:37:03.377125 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.376880    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp6g4\" (UniqueName: \"kubernetes.io/projected/3ac8ac4b-fef2-475d-a40b-9db77cded4d3-kube-api-access-jp6g4\") pod \"dns-default-868lx\" (UID: \"3ac8ac4b-fef2-475d-a40b-9db77cded4d3\") " pod="openshift-dns/dns-default-868lx"
Nov 03 19:37:03.377125 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.376907    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8771d72e-12a3-429f-b458-e22298238393-crio-socket\") pod \"insights-runtime-extractor-2224c\" (UID: \"8771d72e-12a3-429f-b458-e22298238393\") " pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:03.377125 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.376949    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ac8ac4b-fef2-475d-a40b-9db77cded4d3-metrics-tls\") pod \"dns-default-868lx\" (UID: \"3ac8ac4b-fef2-475d-a40b-9db77cded4d3\") " pod="openshift-dns/dns-default-868lx"
Nov 03 19:37:03.377125 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.377003    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8771d72e-12a3-429f-b458-e22298238393-data-volume\") pod \"insights-runtime-extractor-2224c\" (UID: \"8771d72e-12a3-429f-b458-e22298238393\") " pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:03.377125 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.377072    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ac8ac4b-fef2-475d-a40b-9db77cded4d3-tmp-dir\") pod \"dns-default-868lx\" (UID: \"3ac8ac4b-fef2-475d-a40b-9db77cded4d3\") " pod="openshift-dns/dns-default-868lx"
Nov 03 19:37:03.377443 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.377142    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8771d72e-12a3-429f-b458-e22298238393-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2224c\" (UID: \"8771d72e-12a3-429f-b458-e22298238393\") " pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:03.377443 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.377175    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8771d72e-12a3-429f-b458-e22298238393-crio-socket\") pod \"insights-runtime-extractor-2224c\" (UID: \"8771d72e-12a3-429f-b458-e22298238393\") " pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:03.377443 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.377178    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8771d72e-12a3-429f-b458-e22298238393-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2224c\" (UID: \"8771d72e-12a3-429f-b458-e22298238393\") " pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:03.377443 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.377289    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac8ac4b-fef2-475d-a40b-9db77cded4d3-config-volume\") pod \"dns-default-868lx\" (UID: \"3ac8ac4b-fef2-475d-a40b-9db77cded4d3\") " pod="openshift-dns/dns-default-868lx"
Nov 03 19:37:03.377443 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.377355    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8771d72e-12a3-429f-b458-e22298238393-data-volume\") pod \"insights-runtime-extractor-2224c\" (UID: \"8771d72e-12a3-429f-b458-e22298238393\") " pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:03.377757 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.377738    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8771d72e-12a3-429f-b458-e22298238393-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2224c\" (UID: \"8771d72e-12a3-429f-b458-e22298238393\") " pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:03.377842 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.377822    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac8ac4b-fef2-475d-a40b-9db77cded4d3-config-volume\") pod \"dns-default-868lx\" (UID: \"3ac8ac4b-fef2-475d-a40b-9db77cded4d3\") " pod="openshift-dns/dns-default-868lx"
Nov 03 19:37:03.380864 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.380800    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ac8ac4b-fef2-475d-a40b-9db77cded4d3-metrics-tls\") pod \"dns-default-868lx\" (UID: \"3ac8ac4b-fef2-475d-a40b-9db77cded4d3\") " pod="openshift-dns/dns-default-868lx"
Nov 03 19:37:03.380864 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.380812    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8771d72e-12a3-429f-b458-e22298238393-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2224c\" (UID: \"8771d72e-12a3-429f-b458-e22298238393\") " pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:03.381007 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.380901    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f926eeff-6488-48c5-9e97-269a6d79dee1-cert\") pod \"ingress-canary-mksqw\" (UID: \"f926eeff-6488-48c5-9e97-269a6d79dee1\") " pod="openshift-ingress-canary/ingress-canary-mksqw"
Nov 03 19:37:03.389470 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.389441    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp6g4\" (UniqueName: \"kubernetes.io/projected/3ac8ac4b-fef2-475d-a40b-9db77cded4d3-kube-api-access-jp6g4\") pod \"dns-default-868lx\" (UID: \"3ac8ac4b-fef2-475d-a40b-9db77cded4d3\") " pod="openshift-dns/dns-default-868lx"
Nov 03 19:37:03.389770 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.389750    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4545\" (UniqueName: \"kubernetes.io/projected/f926eeff-6488-48c5-9e97-269a6d79dee1-kube-api-access-x4545\") pod \"ingress-canary-mksqw\" (UID: \"f926eeff-6488-48c5-9e97-269a6d79dee1\") " pod="openshift-ingress-canary/ingress-canary-mksqw"
Nov 03 19:37:03.390766 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.390744    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sgbt\" (UniqueName: \"kubernetes.io/projected/8771d72e-12a3-429f-b458-e22298238393-kube-api-access-5sgbt\") pod \"insights-runtime-extractor-2224c\" (UID: \"8771d72e-12a3-429f-b458-e22298238393\") " pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:03.490042 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.489989    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mksqw"
Nov 03 19:37:03.498560 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.498544    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:03.518507 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.518466    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-868lx"
Nov 03 19:37:03.781320 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.781289    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mksqw"]
Nov 03 19:37:03.785122 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.785099    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2224c"]
Nov 03 19:37:03.785302 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:37:03.785280    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf926eeff_6488_48c5_9e97_269a6d79dee1.slice/crio-57562e83005ea47370f2c057b8116aa6f5e7fd92322963cdc16c9849d4ca87cd WatchSource:0}: Error finding container 57562e83005ea47370f2c057b8116aa6f5e7fd92322963cdc16c9849d4ca87cd: Status 404 returned error can't find the container with id 57562e83005ea47370f2c057b8116aa6f5e7fd92322963cdc16c9849d4ca87cd
Nov 03 19:37:03.787945 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:37:03.787922    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8771d72e_12a3_429f_b458_e22298238393.slice/crio-62f8eb6fd8dcd0e75732103842fb7277ccb2358ace74075dc8fa03dc3a13af04 WatchSource:0}: Error finding container 62f8eb6fd8dcd0e75732103842fb7277ccb2358ace74075dc8fa03dc3a13af04: Status 404 returned error can't find the container with id 62f8eb6fd8dcd0e75732103842fb7277ccb2358ace74075dc8fa03dc3a13af04
Nov 03 19:37:03.821355 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.821331    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-868lx"]
Nov 03 19:37:03.824353 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:37:03.824328    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac8ac4b_fef2_475d_a40b_9db77cded4d3.slice/crio-0eaf6115f4579620b0f14c3598a9d52a4cd6355129bde16bac809b7d568be2ec WatchSource:0}: Error finding container 0eaf6115f4579620b0f14c3598a9d52a4cd6355129bde16bac809b7d568be2ec: Status 404 returned error can't find the container with id 0eaf6115f4579620b0f14c3598a9d52a4cd6355129bde16bac809b7d568be2ec
Nov 03 19:37:03.843925 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.843895    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:37:03.848600 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.848575    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tx8mk\""
Nov 03 19:37:03.848907 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:03.848886    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 19:37:04.091023 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.090946    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-868lx" event={"ID":"3ac8ac4b-fef2-475d-a40b-9db77cded4d3","Type":"ContainerStarted","Data":"0eaf6115f4579620b0f14c3598a9d52a4cd6355129bde16bac809b7d568be2ec"}
Nov 03 19:37:04.092381 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.092358    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2224c" event={"ID":"8771d72e-12a3-429f-b458-e22298238393","Type":"ContainerStarted","Data":"3d53ace65bc2ebb532a55208ee79c79c9ab9b258e063e376e58d8ebe53ea3fe8"}
Nov 03 19:37:04.092443 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.092412    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2224c" event={"ID":"8771d72e-12a3-429f-b458-e22298238393","Type":"ContainerStarted","Data":"62f8eb6fd8dcd0e75732103842fb7277ccb2358ace74075dc8fa03dc3a13af04"}
Nov 03 19:37:04.093408 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.093349    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mksqw" event={"ID":"f926eeff-6488-48c5-9e97-269a6d79dee1","Type":"ContainerStarted","Data":"57562e83005ea47370f2c057b8116aa6f5e7fd92322963cdc16c9849d4ca87cd"}
Nov 03 19:37:04.467997 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.467914    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rvqts"]
Nov 03 19:37:04.470781 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.470084    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rvqts"
Nov 03 19:37:04.474827 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.474808    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-xc2tq\""
Nov 03 19:37:04.477362 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.477339    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 03 19:37:04.480904 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.480885    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 19:37:04.481886 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.481867    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rvqts"]
Nov 03 19:37:04.482244 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.482225    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 19:37:04.482925 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.482906    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 19:37:04.562143 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.562122    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 03 19:37:04.588490 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.588464    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1cd0db81-8027-4c6a-b33c-3b4545351eac-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rvqts\" (UID: \"1cd0db81-8027-4c6a-b33c-3b4545351eac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rvqts"
Nov 03 19:37:04.588613 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.588541    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1cd0db81-8027-4c6a-b33c-3b4545351eac-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rvqts\" (UID: \"1cd0db81-8027-4c6a-b33c-3b4545351eac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rvqts"
Nov 03 19:37:04.588689 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.588652    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1cd0db81-8027-4c6a-b33c-3b4545351eac-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rvqts\" (UID: \"1cd0db81-8027-4c6a-b33c-3b4545351eac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rvqts"
Nov 03 19:37:04.588760 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.588731    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvddx\" (UniqueName: \"kubernetes.io/projected/1cd0db81-8027-4c6a-b33c-3b4545351eac-kube-api-access-gvddx\") pod \"prometheus-operator-9cd89b74c-rvqts\" (UID: \"1cd0db81-8027-4c6a-b33c-3b4545351eac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rvqts"
Nov 03 19:37:04.689438 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.689403    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1cd0db81-8027-4c6a-b33c-3b4545351eac-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rvqts\" (UID: \"1cd0db81-8027-4c6a-b33c-3b4545351eac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rvqts"
Nov 03 19:37:04.689600 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.689491    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1cd0db81-8027-4c6a-b33c-3b4545351eac-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rvqts\" (UID: \"1cd0db81-8027-4c6a-b33c-3b4545351eac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rvqts"
Nov 03 19:37:04.689600 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.689541    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1cd0db81-8027-4c6a-b33c-3b4545351eac-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rvqts\" (UID: \"1cd0db81-8027-4c6a-b33c-3b4545351eac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rvqts"
Nov 03 19:37:04.689600 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.689572    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvddx\" (UniqueName: \"kubernetes.io/projected/1cd0db81-8027-4c6a-b33c-3b4545351eac-kube-api-access-gvddx\") pod \"prometheus-operator-9cd89b74c-rvqts\" (UID: \"1cd0db81-8027-4c6a-b33c-3b4545351eac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rvqts"
Nov 03 19:37:04.690823 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.690791    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1cd0db81-8027-4c6a-b33c-3b4545351eac-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rvqts\" (UID: \"1cd0db81-8027-4c6a-b33c-3b4545351eac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rvqts"
Nov 03 19:37:04.692530 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.692483    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1cd0db81-8027-4c6a-b33c-3b4545351eac-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rvqts\" (UID: \"1cd0db81-8027-4c6a-b33c-3b4545351eac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rvqts"
Nov 03 19:37:04.696986 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.696963    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1cd0db81-8027-4c6a-b33c-3b4545351eac-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rvqts\" (UID: \"1cd0db81-8027-4c6a-b33c-3b4545351eac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rvqts"
Nov 03 19:37:04.703400 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.703381    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvddx\" (UniqueName: \"kubernetes.io/projected/1cd0db81-8027-4c6a-b33c-3b4545351eac-kube-api-access-gvddx\") pod \"prometheus-operator-9cd89b74c-rvqts\" (UID: \"1cd0db81-8027-4c6a-b33c-3b4545351eac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rvqts"
Nov 03 19:37:04.787573 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.787542    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rvqts"
Nov 03 19:37:04.843813 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.843778    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:37:04.843952 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.843828    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:37:04.853713 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.853690    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 19:37:04.854798 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.853836    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5qm8x\""
Nov 03 19:37:04.854798 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.853887    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 19:37:04.854798 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.853945    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 19:37:04.854798 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:04.854447    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vlmgx\""
Nov 03 19:37:05.073160 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:05.073081    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rvqts"]
Nov 03 19:37:05.080468 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:37:05.080439    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd0db81_8027_4c6a_b33c_3b4545351eac.slice/crio-8c27ff198d55b17c368881572802b452da4123f9e871a6827814ff5a986e86b8 WatchSource:0}: Error finding container 8c27ff198d55b17c368881572802b452da4123f9e871a6827814ff5a986e86b8: Status 404 returned error can't find the container with id 8c27ff198d55b17c368881572802b452da4123f9e871a6827814ff5a986e86b8
Nov 03 19:37:05.098607 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:05.098573    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2224c" event={"ID":"8771d72e-12a3-429f-b458-e22298238393","Type":"ContainerStarted","Data":"f3002e9582ae24606911cc32c20e33db5ba571a0ff7f54f0cd8910febd6c699c"}
Nov 03 19:37:05.099992 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:05.099962    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rvqts" event={"ID":"1cd0db81-8027-4c6a-b33c-3b4545351eac","Type":"ContainerStarted","Data":"8c27ff198d55b17c368881572802b452da4123f9e871a6827814ff5a986e86b8"}
Nov 03 19:37:06.110156 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:06.110117    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-868lx" event={"ID":"3ac8ac4b-fef2-475d-a40b-9db77cded4d3","Type":"ContainerStarted","Data":"f54212cc6287c4807e6d92a8530825f9258aa714156867d4ec7d41268b42b6c1"}
Nov 03 19:37:06.110578 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:06.110188    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-868lx" event={"ID":"3ac8ac4b-fef2-475d-a40b-9db77cded4d3","Type":"ContainerStarted","Data":"4e55200283e77ca516c90bf63a1e13a17ef48fdd6873a76c2dba75bc4bb2b950"}
Nov 03 19:37:06.110898 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:06.110875    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-868lx"
Nov 03 19:37:06.132454 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:06.132408    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-868lx" podStartSLOduration=1.268138831 podStartE2EDuration="3.132393394s" totalImagesPullingTime="1.864254563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:06.131571064 +0000 UTC m=+46.776761394" watchObservedRunningTime="2026-11-03 19:37:06.132393394 +0000 UTC m=+46.777583694"
Nov 03 19:37:07.114843 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:07.114802    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mksqw" event={"ID":"f926eeff-6488-48c5-9e97-269a6d79dee1","Type":"ContainerStarted","Data":"43e2093eab6fcbd4fc94b3a144aff5aa1f6fffdddd5aedc9842b6b7ed90123a6"}
Nov 03 19:37:07.116874 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:07.116852    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2224c" event={"ID":"8771d72e-12a3-429f-b458-e22298238393","Type":"ContainerStarted","Data":"7b3aa0486897fb352e01ea0065b38ac0b5473d036ae527768dfb209a0518dac3"}
Nov 03 19:37:07.118468 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:07.118447    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rvqts" event={"ID":"1cd0db81-8027-4c6a-b33c-3b4545351eac","Type":"ContainerStarted","Data":"c7f46002282f0441ba93cc1edc99269d704a7d53c546d2b4806bb48eb9404f89"}
Nov 03 19:37:07.118583 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:07.118482    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rvqts" event={"ID":"1cd0db81-8027-4c6a-b33c-3b4545351eac","Type":"ContainerStarted","Data":"b8282900d5ee6a62fba2bd04fe5ad251028e0ca87a366b2ccd7be0df58e1c0b2"}
Nov 03 19:37:07.133987 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:07.133947    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mksqw" podStartSLOduration=1.18400703 podStartE2EDuration="4.133936186s" totalImagesPullingTime="2.949929156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:07.133338391 +0000 UTC m=+47.778528693" watchObservedRunningTime="2026-11-03 19:37:07.133936186 +0000 UTC m=+47.779126488"
Nov 03 19:37:07.154640 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:07.154598    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2224c" podStartSLOduration=1.7838069079999999 podStartE2EDuration="4.154581275s" totalImagesPullingTime="2.370774367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:37:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:07.153844092 +0000 UTC m=+47.799034397" watchObservedRunningTime="2026-11-03 19:37:07.154581275 +0000 UTC m=+47.799771578"
Nov 03 19:37:07.173986 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:07.173927    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rvqts" podStartSLOduration=1.757643926 podStartE2EDuration="3.173909567s" totalImagesPullingTime="1.416265641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:07.172945467 +0000 UTC m=+47.818135770" watchObservedRunningTime="2026-11-03 19:37:07.173909567 +0000 UTC m=+47.819099871"
Nov 03 19:37:08.865406 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:08.865375    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"]
Nov 03 19:37:08.867453 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:08.867438    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:08.885989 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:08.885963    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"]
Nov 03 19:37:08.893502 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:08.893480    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sckcx"]
Nov 03 19:37:08.895451 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:08.895437    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:08.928880 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:08.928840    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8hqp\" (UniqueName: \"kubernetes.io/projected/300a6502-6932-49fe-936a-ba0c16978c94-kube-api-access-x8hqp\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:08.929028 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:08.928886    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/300a6502-6932-49fe-936a-ba0c16978c94-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:08.929028 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:08.929004    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/300a6502-6932-49fe-936a-ba0c16978c94-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:08.929103 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:08.929037    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/300a6502-6932-49fe-936a-ba0c16978c94-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:08.929103 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:08.929059    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/300a6502-6932-49fe-936a-ba0c16978c94-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:08.929194 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:08.929103    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/300a6502-6932-49fe-936a-ba0c16978c94-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:08.949973 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:08.949942    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 19:37:08.949973 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:08.949962    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 03 19:37:08.950305 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:08.950153    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lhxcg\""
Nov 03 19:37:08.950305 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:08.950180    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 03 19:37:08.950305 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:08.950270    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 03 19:37:08.950305 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:08.950305    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 19:37:08.950439 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:08.950361    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-54rzd\""
Nov 03 19:37:08.951506 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:08.951492    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 19:37:09.030363 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.030326    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/803910f1-a1f3-4316-ada9-1890e46bbda1-metrics-client-ca\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.030528 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.030378    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/300a6502-6932-49fe-936a-ba0c16978c94-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:09.030528 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.030408    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/803910f1-a1f3-4316-ada9-1890e46bbda1-sys\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.030528 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.030467    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-wtmp\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.030623 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.030549    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/300a6502-6932-49fe-936a-ba0c16978c94-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:09.030655 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.030631    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-textfile\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.030702 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.030682    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.030735 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.030716    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/803910f1-a1f3-4316-ada9-1890e46bbda1-root\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.030778 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.030743    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-tls\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.030837 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.030820    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/300a6502-6932-49fe-936a-ba0c16978c94-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:09.030871 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.030859    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8hqp\" (UniqueName: \"kubernetes.io/projected/300a6502-6932-49fe-936a-ba0c16978c94-kube-api-access-x8hqp\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:09.030903 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.030881    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-accelerators-collector-config\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.030953 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.030921    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/300a6502-6932-49fe-936a-ba0c16978c94-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:09.031128 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.031108    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/300a6502-6932-49fe-936a-ba0c16978c94-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:09.031198 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.031142    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6rwg\" (UniqueName: \"kubernetes.io/projected/803910f1-a1f3-4316-ada9-1890e46bbda1-kube-api-access-q6rwg\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.031287 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.031197    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/300a6502-6932-49fe-936a-ba0c16978c94-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:09.031287 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.031250    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/300a6502-6932-49fe-936a-ba0c16978c94-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:09.031983 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.031963    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/300a6502-6932-49fe-936a-ba0c16978c94-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:09.033354 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.033332    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/300a6502-6932-49fe-936a-ba0c16978c94-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:09.033432 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.033375    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/300a6502-6932-49fe-936a-ba0c16978c94-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:09.043318 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.043300    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8hqp\" (UniqueName: \"kubernetes.io/projected/300a6502-6932-49fe-936a-ba0c16978c94-kube-api-access-x8hqp\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:09.132273 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.132177    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.132273 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.132228    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/803910f1-a1f3-4316-ada9-1890e46bbda1-root\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.132273 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.132247    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-tls\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.132273 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.132272    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-accelerators-collector-config\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.132565 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.132309    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6rwg\" (UniqueName: \"kubernetes.io/projected/803910f1-a1f3-4316-ada9-1890e46bbda1-kube-api-access-q6rwg\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.132565 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.132312    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/803910f1-a1f3-4316-ada9-1890e46bbda1-root\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.132565 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.132343    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/803910f1-a1f3-4316-ada9-1890e46bbda1-metrics-client-ca\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.132565 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.132523    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/803910f1-a1f3-4316-ada9-1890e46bbda1-sys\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.132565 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.132564    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-wtmp\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.132829 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.132593    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-textfile\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.132829 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.132638    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/803910f1-a1f3-4316-ada9-1890e46bbda1-sys\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.132829 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.132754    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-wtmp\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.132969 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.132940    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-textfile\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.133066 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.133051    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/803910f1-a1f3-4316-ada9-1890e46bbda1-metrics-client-ca\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.133141 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.133118    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-accelerators-collector-config\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.134669 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.134653    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.134768 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.134744    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-tls\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.144552 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.144532    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6rwg\" (UniqueName: \"kubernetes.io/projected/803910f1-a1f3-4316-ada9-1890e46bbda1-kube-api-access-q6rwg\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.179330 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.179302    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:09.230056 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.229855    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:09.241187 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:37:09.241138    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod803910f1_a1f3_4316_ada9_1890e46bbda1.slice/crio-c3e81c5328eec5ff651a59728145ec14d13cc2e0bd31f41d97f17321fae9a3e3 WatchSource:0}: Error finding container c3e81c5328eec5ff651a59728145ec14d13cc2e0bd31f41d97f17321fae9a3e3: Status 404 returned error can't find the container with id c3e81c5328eec5ff651a59728145ec14d13cc2e0bd31f41d97f17321fae9a3e3
Nov 03 19:37:09.451669 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.451554    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"]
Nov 03 19:37:09.454042 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:37:09.453991    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod300a6502_6932_49fe_936a_ba0c16978c94.slice/crio-5672d069f537ac87a1d7070bb4a617199f7a10c736d6206a803d2a3273952c9a WatchSource:0}: Error finding container 5672d069f537ac87a1d7070bb4a617199f7a10c736d6206a803d2a3273952c9a: Status 404 returned error can't find the container with id 5672d069f537ac87a1d7070bb4a617199f7a10c736d6206a803d2a3273952c9a
Nov 03 19:37:09.970216 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.970183    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:37:09.972757 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.972737    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:09.977777 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.977753    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 19:37:09.977901 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.977780    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 19:37:09.977901 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.977856    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 19:37:09.977901 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.977871    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 19:37:09.978061 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.977909    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 19:37:09.978061 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.977946    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 19:37:09.978151 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.978075    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pz944\""
Nov 03 19:37:09.978679 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.978652    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 19:37:09.978679 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.978652    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 19:37:09.978891 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.978702    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 19:37:09.995767 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:09.995673    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:37:10.038986 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.038948    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-config-out\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.038986 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.038999    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.039259 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.039119    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.039259 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.039163    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.039351 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.039263    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.039351 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.039325    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-config-volume\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.039412 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.039379    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.039444 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.039430    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.039509 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.039488    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.039565 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.039525    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-web-config\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.039565 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.039550    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmggx\" (UniqueName: \"kubernetes.io/projected/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-kube-api-access-tmggx\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.039649 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.039635    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.039689 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.039660    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.131264 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.131221    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sckcx" event={"ID":"803910f1-a1f3-4316-ada9-1890e46bbda1","Type":"ContainerStarted","Data":"c3e81c5328eec5ff651a59728145ec14d13cc2e0bd31f41d97f17321fae9a3e3"}
Nov 03 19:37:10.132712 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.132670    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk" event={"ID":"300a6502-6932-49fe-936a-ba0c16978c94","Type":"ContainerStarted","Data":"5672d069f537ac87a1d7070bb4a617199f7a10c736d6206a803d2a3273952c9a"}
Nov 03 19:37:10.140355 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.140320    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-config-volume\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.140515 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.140364    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.140515 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.140408    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.140515 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.140431    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.140515 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.140452    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-web-config\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.140515 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.140478    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmggx\" (UniqueName: \"kubernetes.io/projected/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-kube-api-access-tmggx\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.140515 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.140516    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.140806 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.140539    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.140806 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.140591    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-config-out\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.140806 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.140613    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.140806 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.140635    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.140806 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.140660    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.140806 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.140706    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.142232 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.142161    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.142401 ip-10-0-143-206 kubenswrapper[2566]: E1103 19:37:10.142381    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-alertmanager-trusted-ca-bundle podName:8b9ebd2a-44e9-41f6-8bef-c41257c9a163 nodeName:}" failed. No retries permitted until 2026-11-03 19:37:10.642362161 +0000 UTC m=+51.287552462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "8b9ebd2a-44e9-41f6-8bef-c41257c9a163") : configmap references non-existent config key: ca-bundle.crt
Nov 03 19:37:10.143889 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.143866    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.144228 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.144169    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-config-volume\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.144465 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.144444    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.144535 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.144519    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.144584 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.144566    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.145619 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.145583    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-config-out\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.145710 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.145619    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.145710 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.145643    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.147279 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.147236    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-web-config\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.147487 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.147466    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.156093 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.156067    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmggx\" (UniqueName: \"kubernetes.io/projected/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-kube-api-access-tmggx\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.648158 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.648115    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.649132 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.649103    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.893569 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.893392    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:10.958092 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.956354    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"]
Nov 03 19:37:10.967572 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.967546    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:10.981932 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.981908    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"]
Nov 03 19:37:10.989792 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.989738    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 19:37:10.989792 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.989771    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 19:37:10.989993 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.989854    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 19:37:10.989993 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.989913    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-53fusnobhruq3\""
Nov 03 19:37:10.989993 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.989921    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 19:37:10.989993 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.989928    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-nxvmw\""
Nov 03 19:37:10.989993 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:10.989954    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 19:37:11.051828 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.051801    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-grpc-tls\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.051956 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.051835    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n2qg\" (UniqueName: \"kubernetes.io/projected/c7e50fe5-0dd0-4f59-bc5f-99199500790b-kube-api-access-7n2qg\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.051956 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.051854    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7e50fe5-0dd0-4f59-bc5f-99199500790b-metrics-client-ca\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.051956 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.051929    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.052065 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.052028    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7e50fe5-0dd0-4f59-bc5f-99199500790b-tmp\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.052127 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.052112    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.052163 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.052152    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-tls\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.052225 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.052196    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.052296 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.052280    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.141366 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.141327    2566 generic.go:356] "Generic (PLEG): container finished" podID="803910f1-a1f3-4316-ada9-1890e46bbda1" containerID="17c788662e3bd5d72eef80a87bb40ca5ea53b11c218d5473cd2aeee2274dbcd3" exitCode=0
Nov 03 19:37:11.141531 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.141415    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sckcx" event={"ID":"803910f1-a1f3-4316-ada9-1890e46bbda1","Type":"ContainerDied","Data":"17c788662e3bd5d72eef80a87bb40ca5ea53b11c218d5473cd2aeee2274dbcd3"}
Nov 03 19:37:11.143727 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.143699    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk" event={"ID":"300a6502-6932-49fe-936a-ba0c16978c94","Type":"ContainerStarted","Data":"4ba5edf0c26895b6a8a502beb1fbb05c470a1ef01299bf40fab62720b48f7a31"}
Nov 03 19:37:11.143835 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.143756    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk" event={"ID":"300a6502-6932-49fe-936a-ba0c16978c94","Type":"ContainerStarted","Data":"ed02bb16623e8e020a6131a0414ba18e79f464c907ca031c50461492d03d5a1d"}
Nov 03 19:37:11.143835 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.143784    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk" event={"ID":"300a6502-6932-49fe-936a-ba0c16978c94","Type":"ContainerStarted","Data":"258fb5bab6ce7c159f30be28dd894d2f77727a54d52f5e97edc2d64e9112a0cb"}
Nov 03 19:37:11.153195 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.153169    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.153315 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.153250    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-tls\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.153315 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.153304    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.153424 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.153352    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.153424 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.153387    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-grpc-tls\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.153424 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.153419    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n2qg\" (UniqueName: \"kubernetes.io/projected/c7e50fe5-0dd0-4f59-bc5f-99199500790b-kube-api-access-7n2qg\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.153568 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.153449    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7e50fe5-0dd0-4f59-bc5f-99199500790b-metrics-client-ca\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.153568 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.153489    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.153568 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.153517    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7e50fe5-0dd0-4f59-bc5f-99199500790b-tmp\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.153932 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.153909    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7e50fe5-0dd0-4f59-bc5f-99199500790b-tmp\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.154988 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.154963    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7e50fe5-0dd0-4f59-bc5f-99199500790b-metrics-client-ca\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.156441 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.156417    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.156441 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.156433    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.157889 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.157864    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.158157 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.158135    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-grpc-tls\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.159477 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.159452    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.166464 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.166440    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n2qg\" (UniqueName: \"kubernetes.io/projected/c7e50fe5-0dd0-4f59-bc5f-99199500790b-kube-api-access-7n2qg\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.167245 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.167227    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-tls\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.185096 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.185046    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk" podStartSLOduration=1.9381379 podStartE2EDuration="3.185031081s" totalImagesPullingTime="1.246893181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:11.184567362 +0000 UTC m=+51.829757682" watchObservedRunningTime="2026-11-03 19:37:11.185031081 +0000 UTC m=+51.830221381"
Nov 03 19:37:11.193841 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.193813    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:37:11.198518 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:37:11.198491    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9ebd2a_44e9_41f6_8bef_c41257c9a163.slice/crio-77de5d1cd3a6ba081b6080676f08c8275491705c308f21e922697ae3ed8f06dc WatchSource:0}: Error finding container 77de5d1cd3a6ba081b6080676f08c8275491705c308f21e922697ae3ed8f06dc: Status 404 returned error can't find the container with id 77de5d1cd3a6ba081b6080676f08c8275491705c308f21e922697ae3ed8f06dc
Nov 03 19:37:11.286070 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.286051    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:11.569525 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:11.569496    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"]
Nov 03 19:37:11.572220 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:37:11.572175    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7e50fe5_0dd0_4f59_bc5f_99199500790b.slice/crio-48770e51f689ef669cbfb99db1b67bb09f19a6ca313a5d7c90707b36979f2dc2 WatchSource:0}: Error finding container 48770e51f689ef669cbfb99db1b67bb09f19a6ca313a5d7c90707b36979f2dc2: Status 404 returned error can't find the container with id 48770e51f689ef669cbfb99db1b67bb09f19a6ca313a5d7c90707b36979f2dc2
Nov 03 19:37:12.149122 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:12.149020    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b9ebd2a-44e9-41f6-8bef-c41257c9a163","Type":"ContainerStarted","Data":"77de5d1cd3a6ba081b6080676f08c8275491705c308f21e922697ae3ed8f06dc"}
Nov 03 19:37:12.150636 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:12.150593    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct" event={"ID":"c7e50fe5-0dd0-4f59-bc5f-99199500790b","Type":"ContainerStarted","Data":"48770e51f689ef669cbfb99db1b67bb09f19a6ca313a5d7c90707b36979f2dc2"}
Nov 03 19:37:12.154651 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:12.154607    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sckcx" event={"ID":"803910f1-a1f3-4316-ada9-1890e46bbda1","Type":"ContainerStarted","Data":"683d25d54e154a0495aed28482fe85fa9c7a45719bd000e3b00ab2018a97684a"}
Nov 03 19:37:12.154802 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:12.154654    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sckcx" event={"ID":"803910f1-a1f3-4316-ada9-1890e46bbda1","Type":"ContainerStarted","Data":"5271db6045c9c877a25a45a3a2aaef767055d4449deb0478f5e8539dfb2dbc45"}
Nov 03 19:37:12.177092 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:12.177031    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sckcx" podStartSLOduration=2.8322474189999998 podStartE2EDuration="4.177014607s" totalImagesPullingTime="1.257062015s" totalInitContainerRuntime="87.705173ms" isStatefulPod=true podCreationTimestamp="2026-11-03 19:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:12.175890724 +0000 UTC m=+52.821081028" watchObservedRunningTime="2026-11-03 19:37:12.177014607 +0000 UTC m=+52.822204912"
Nov 03 19:37:13.158416 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:13.158381    2566 generic.go:356] "Generic (PLEG): container finished" podID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerID="5d900c564cd425ff4cebb7144fdc0d60bf5953d447cc834619b884d13dce2b77" exitCode=0
Nov 03 19:37:13.158863 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:13.158465    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b9ebd2a-44e9-41f6-8bef-c41257c9a163","Type":"ContainerDied","Data":"5d900c564cd425ff4cebb7144fdc0d60bf5953d447cc834619b884d13dce2b77"}
Nov 03 19:37:13.657508 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:13.657456    2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-bnrmk"]
Nov 03 19:37:13.659854 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:13.659835    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bnrmk"
Nov 03 19:37:13.663648 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:13.663629    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 19:37:13.664549 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:13.664506    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5x9cb\""
Nov 03 19:37:13.670368 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:13.670280    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-bnrmk"]
Nov 03 19:37:13.777519 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:13.777482    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f6bc2cf5-3e45-4217-af83-38df92cfee7f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-bnrmk\" (UID: \"f6bc2cf5-3e45-4217-af83-38df92cfee7f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bnrmk"
Nov 03 19:37:13.777519 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:13.777518    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz2hs\" (UniqueName: \"kubernetes.io/projected/f6bc2cf5-3e45-4217-af83-38df92cfee7f-kube-api-access-rz2hs\") pod \"monitoring-plugin-77f4f76594-bnrmk\" (UID: \"f6bc2cf5-3e45-4217-af83-38df92cfee7f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bnrmk"
Nov 03 19:37:13.878690 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:13.878652    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f6bc2cf5-3e45-4217-af83-38df92cfee7f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-bnrmk\" (UID: \"f6bc2cf5-3e45-4217-af83-38df92cfee7f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bnrmk"
Nov 03 19:37:13.878863 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:13.878698    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz2hs\" (UniqueName: \"kubernetes.io/projected/f6bc2cf5-3e45-4217-af83-38df92cfee7f-kube-api-access-rz2hs\") pod \"monitoring-plugin-77f4f76594-bnrmk\" (UID: \"f6bc2cf5-3e45-4217-af83-38df92cfee7f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bnrmk"
Nov 03 19:37:13.881717 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:13.881688    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f6bc2cf5-3e45-4217-af83-38df92cfee7f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-bnrmk\" (UID: \"f6bc2cf5-3e45-4217-af83-38df92cfee7f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bnrmk"
Nov 03 19:37:13.899242 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:13.899194    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz2hs\" (UniqueName: \"kubernetes.io/projected/f6bc2cf5-3e45-4217-af83-38df92cfee7f-kube-api-access-rz2hs\") pod \"monitoring-plugin-77f4f76594-bnrmk\" (UID: \"f6bc2cf5-3e45-4217-af83-38df92cfee7f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bnrmk"
Nov 03 19:37:13.975164 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:13.975086    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bnrmk"
Nov 03 19:37:14.064272 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:14.064106    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:14.165959 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:14.165924    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct" event={"ID":"c7e50fe5-0dd0-4f59-bc5f-99199500790b","Type":"ContainerStarted","Data":"71fb2e7b503b9c084c8a62923a34f0d8b7fa8f9781a6f5c33edcf79f3a138a2d"}
Nov 03 19:37:14.166456 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:14.165990    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct" event={"ID":"c7e50fe5-0dd0-4f59-bc5f-99199500790b","Type":"ContainerStarted","Data":"446fbd428b3a8e64ba0a6276b73e594ec11928a062ad5d3a96e9a572a013ceba"}
Nov 03 19:37:14.166456 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:14.166017    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct" event={"ID":"c7e50fe5-0dd0-4f59-bc5f-99199500790b","Type":"ContainerStarted","Data":"b85c9ca14cc1b5adc60aa45990d8c5192b57a07efca2b67d18bf2465cd004789"}
Nov 03 19:37:14.255048 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:14.254604    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-bnrmk"]
Nov 03 19:37:14.257942 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:37:14.257911    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6bc2cf5_3e45_4217_af83_38df92cfee7f.slice/crio-0f241ca48f494c140d074ff6fcfb46095a2ca4874c8c33c1d31069343b99f3dd WatchSource:0}: Error finding container 0f241ca48f494c140d074ff6fcfb46095a2ca4874c8c33c1d31069343b99f3dd: Status 404 returned error can't find the container with id 0f241ca48f494c140d074ff6fcfb46095a2ca4874c8c33c1d31069343b99f3dd
Nov 03 19:37:15.180685 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:15.180426    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct" event={"ID":"c7e50fe5-0dd0-4f59-bc5f-99199500790b","Type":"ContainerStarted","Data":"1655fb846936b0d7c96b81f185b3c1df7d930b81018202be3944fdbbd40436ae"}
Nov 03 19:37:15.180685 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:15.180508    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct" event={"ID":"c7e50fe5-0dd0-4f59-bc5f-99199500790b","Type":"ContainerStarted","Data":"616317aaa2bc58ea4636a28a374b4a67271be17c16a92e95c6a5915140f91128"}
Nov 03 19:37:15.180685 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:15.180550    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct" event={"ID":"c7e50fe5-0dd0-4f59-bc5f-99199500790b","Type":"ContainerStarted","Data":"e8b149a10f3ce98ba8c2e9a2420524a215028cd57cb5c37c735d00b4cc6c0463"}
Nov 03 19:37:15.180685 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:15.180627    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:15.184162 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:15.184134    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bnrmk" event={"ID":"f6bc2cf5-3e45-4217-af83-38df92cfee7f","Type":"ContainerStarted","Data":"0f241ca48f494c140d074ff6fcfb46095a2ca4874c8c33c1d31069343b99f3dd"}
Nov 03 19:37:15.193913 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:15.193878    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b9ebd2a-44e9-41f6-8bef-c41257c9a163","Type":"ContainerStarted","Data":"987605c633836498a8cf1f0041052cb35897382152a89641b4af7f70e626d264"}
Nov 03 19:37:15.194161 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:15.194143    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b9ebd2a-44e9-41f6-8bef-c41257c9a163","Type":"ContainerStarted","Data":"fd06f1ac57f7a0e0cf584f9d40064856fe743086cbf9b2b002655cc6c3d62aad"}
Nov 03 19:37:15.194349 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:15.194336    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b9ebd2a-44e9-41f6-8bef-c41257c9a163","Type":"ContainerStarted","Data":"a1f1af32f35471807dd91f0988f71f6fcbedd923261adceee04ba26995263dc6"}
Nov 03 19:37:15.194453 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:15.194444    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b9ebd2a-44e9-41f6-8bef-c41257c9a163","Type":"ContainerStarted","Data":"e281919b8c165d59b742e4aaa3f9e48d836a4437355c03c4ab167cd60bb360bd"}
Nov 03 19:37:15.213117 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:15.211878    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct" podStartSLOduration=2.595139944 podStartE2EDuration="5.211861143s" totalImagesPullingTime="2.616721199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:15.211187857 +0000 UTC m=+55.856378159" watchObservedRunningTime="2026-11-03 19:37:15.211861143 +0000 UTC m=+55.857051450"
Nov 03 19:37:16.200455 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:16.200418    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b9ebd2a-44e9-41f6-8bef-c41257c9a163","Type":"ContainerStarted","Data":"28bba717aa2ceae16309031ea60bbadef31e99a8ffde5698d7ebeec24e02d467"}
Nov 03 19:37:16.200910 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:16.200494    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b9ebd2a-44e9-41f6-8bef-c41257c9a163","Type":"ContainerStarted","Data":"89417e2a306c84e55d86c07a221fb764a986678e3d257bd6c5c7a83d0360df8e"}
Nov 03 19:37:16.201851 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:16.201824    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bnrmk" event={"ID":"f6bc2cf5-3e45-4217-af83-38df92cfee7f","Type":"ContainerStarted","Data":"a3a71ae11a41260e2adb76da49f0cc678b3c1a1403b17652416130f9c8f60c0d"}
Nov 03 19:37:16.202184 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:16.202166    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bnrmk"
Nov 03 19:37:16.207255 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:16.207239    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bnrmk"
Nov 03 19:37:16.235427 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:16.234922    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.514603865 podStartE2EDuration="7.234905706s" totalImagesPullingTime="2.704507764s" totalInitContainerRuntime="15.794077ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:16.232934383 +0000 UTC m=+56.878124687" watchObservedRunningTime="2026-11-03 19:37:16.234905706 +0000 UTC m=+56.880096010"
Nov 03 19:37:16.253040 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:16.253001    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bnrmk" podStartSLOduration=1.7594690530000001 podStartE2EDuration="3.252990438s" totalImagesPullingTime="1.493521385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:16.251872141 +0000 UTC m=+56.897062468" watchObservedRunningTime="2026-11-03 19:37:16.252990438 +0000 UTC m=+56.898180741"
Nov 03 19:37:17.127756 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:17.127730    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-868lx"
Nov 03 19:37:21.208884 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:21.208857    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:25.583314 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:25.583273    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca39ebc9-da71-43c2-893f-c033684c9399-metrics-certs\") pod \"network-metrics-daemon-794nj\" (UID: \"ca39ebc9-da71-43c2-893f-c033684c9399\") " pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:37:25.586482 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:25.586465    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 19:37:25.595934 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:25.595903    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca39ebc9-da71-43c2-893f-c033684c9399-metrics-certs\") pod \"network-metrics-daemon-794nj\" (UID: \"ca39ebc9-da71-43c2-893f-c033684c9399\") " pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:37:25.684313 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:25.684269    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng4gr\" (UniqueName: \"kubernetes.io/projected/f951b540-0179-474c-8a2c-7ee804f9d630-kube-api-access-ng4gr\") pod \"network-check-target-pfcnb\" (UID: \"f951b540-0179-474c-8a2c-7ee804f9d630\") " pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:37:25.684495 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:25.684330    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98d32fbe-5011-47a8-b937-725a8cad95a4-original-pull-secret\") pod \"global-pull-secret-syncer-6q5vq\" (UID: \"98d32fbe-5011-47a8-b937-725a8cad95a4\") " pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:37:25.687622 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:25.687600    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 19:37:25.687622 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:25.687605    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 19:37:25.697199 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:25.697181    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98d32fbe-5011-47a8-b937-725a8cad95a4-original-pull-secret\") pod \"global-pull-secret-syncer-6q5vq\" (UID: \"98d32fbe-5011-47a8-b937-725a8cad95a4\") " pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:37:25.697678 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:25.697665    2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 19:37:25.708047 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:25.708029    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng4gr\" (UniqueName: \"kubernetes.io/projected/f951b540-0179-474c-8a2c-7ee804f9d630-kube-api-access-ng4gr\") pod \"network-check-target-pfcnb\" (UID: \"f951b540-0179-474c-8a2c-7ee804f9d630\") " pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:37:25.758181 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:25.758155    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tx8mk\""
Nov 03 19:37:25.765261 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:25.765240    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:37:25.866376 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:25.866317    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vlmgx\""
Nov 03 19:37:25.874029 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:25.874011    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:37:25.874605 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:25.874591    2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5qm8x\""
Nov 03 19:37:25.881944 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:25.881926    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:37:26.039720 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:26.039694    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-794nj"]
Nov 03 19:37:26.042304 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:37:26.042280    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca39ebc9_da71_43c2_893f_c033684c9399.slice/crio-3e6bc54351c35f29c80caf98aa51d0e66f770cc73a6380359128f72ab58b65d8 WatchSource:0}: Error finding container 3e6bc54351c35f29c80caf98aa51d0e66f770cc73a6380359128f72ab58b65d8: Status 404 returned error can't find the container with id 3e6bc54351c35f29c80caf98aa51d0e66f770cc73a6380359128f72ab58b65d8
Nov 03 19:37:26.154218 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:26.154157    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6q5vq"]
Nov 03 19:37:26.156592 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:37:26.156562    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d32fbe_5011_47a8_b937_725a8cad95a4.slice/crio-d8fa1f9d28b5fd9c66cab82f903427748428dc067259baa487701b7c3e950780 WatchSource:0}: Error finding container d8fa1f9d28b5fd9c66cab82f903427748428dc067259baa487701b7c3e950780: Status 404 returned error can't find the container with id d8fa1f9d28b5fd9c66cab82f903427748428dc067259baa487701b7c3e950780
Nov 03 19:37:26.163691 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:26.163672    2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pfcnb"]
Nov 03 19:37:26.165801 ip-10-0-143-206 kubenswrapper[2566]: W1103 19:37:26.165774    2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf951b540_0179_474c_8a2c_7ee804f9d630.slice/crio-91ac845a4e0fd4d48706806d9296092656f225636a066a05db0d42d15dee5ec3 WatchSource:0}: Error finding container 91ac845a4e0fd4d48706806d9296092656f225636a066a05db0d42d15dee5ec3: Status 404 returned error can't find the container with id 91ac845a4e0fd4d48706806d9296092656f225636a066a05db0d42d15dee5ec3
Nov 03 19:37:26.240163 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:26.240131    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pfcnb" event={"ID":"f951b540-0179-474c-8a2c-7ee804f9d630","Type":"ContainerStarted","Data":"91ac845a4e0fd4d48706806d9296092656f225636a066a05db0d42d15dee5ec3"}
Nov 03 19:37:26.241444 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:26.241419    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-794nj" event={"ID":"ca39ebc9-da71-43c2-893f-c033684c9399","Type":"ContainerStarted","Data":"3e6bc54351c35f29c80caf98aa51d0e66f770cc73a6380359128f72ab58b65d8"}
Nov 03 19:37:26.242581 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:26.242561    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6q5vq" event={"ID":"98d32fbe-5011-47a8-b937-725a8cad95a4","Type":"ContainerStarted","Data":"d8fa1f9d28b5fd9c66cab82f903427748428dc067259baa487701b7c3e950780"}
Nov 03 19:37:28.255664 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:28.255619    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-794nj" event={"ID":"ca39ebc9-da71-43c2-893f-c033684c9399","Type":"ContainerStarted","Data":"40e07e26c7a97f9fb16154a916374e7becd289d49872e350ef3b540ea6a4ca51"}
Nov 03 19:37:28.256135 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:28.255689    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-794nj" event={"ID":"ca39ebc9-da71-43c2-893f-c033684c9399","Type":"ContainerStarted","Data":"53070b81795acf839537b710d41ac6beb65e7e8dc1c9996988d162f8be3d74c9"}
Nov 03 19:37:28.274392 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:28.274340    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-794nj" podStartSLOduration=68.131305975 podStartE2EDuration="1m9.274326586s" totalImagesPullingTime="1.143020611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:28.273461672 +0000 UTC m=+68.918651976" watchObservedRunningTime="2026-11-03 19:37:28.274326586 +0000 UTC m=+68.919516888"
Nov 03 19:37:30.265667 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:30.265622    2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pfcnb" event={"ID":"f951b540-0179-474c-8a2c-7ee804f9d630","Type":"ContainerStarted","Data":"1dce41f3007fd7d31c3cfb1a26f64b6ca04ec9ca84e47c064ccd70fcefb9401a"}
Nov 03 19:37:30.266115 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:30.265766    2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:37:30.285240 ip-10-0-143-206 kubenswrapper[2566]: I1103 19:37:30.285172    2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pfcnb" podStartSLOduration=68.007316881 podStartE2EDuration="1m11.285157381s" totalImagesPullingTime="3.2778405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:36:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:30.284131039 +0000 UTC m=+70.929321348" watchObservedRunningTime="2026-11-03 19:37:30.285157381 +0000 UTC m=+70.930347685"
Nov 03 19:37:31.161943 ip-10-0-143-206 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 19:37:31.168805 ip-10-0-143-206 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 19:37:31.168997 ip-10-0-143-206 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 19:37:31.169052 ip-10-0-143-206 systemd[1]: kubelet.service: Consumed 4.101s CPU time.
Nov 03 19:37:31.184056 ip-10-0-143-206 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 19:37:31.260074 ip-10-0-143-206 kubenswrapper[5591]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:37:31.260074 ip-10-0-143-206 kubenswrapper[5591]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 19:37:31.260074 ip-10-0-143-206 kubenswrapper[5591]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:37:31.260074 ip-10-0-143-206 kubenswrapper[5591]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 19:37:31.262139 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262055    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:37:31.262139 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262140    5591 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:37:31.262139 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262144    5591 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:37:31.262260 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262147    5591 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:37:31.262260 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262151    5591 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:37:31.262260 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262154    5591 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:37:31.262260 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262157    5591 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:37:31.262260 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262160    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:37:31.262260 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262163    5591 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:37:31.262260 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262165    5591 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:37:31.262260 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262168    5591 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:37:31.262260 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262171    5591 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:37:31.262260 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262173    5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:37:31.262260 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262176    5591 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:37:31.262260 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262179    5591 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:37:31.262260 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262181    5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:37:31.262260 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262184    5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:37:31.262260 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262186    5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:37:31.262260 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262189    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:37:31.262260 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262191    5591 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:37:31.262260 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262193    5591 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:37:31.262260 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262196    5591 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:37:31.262260 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262214    5591 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:37:31.262796 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262219    5591 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:37:31.262796 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262222    5591 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:37:31.262796 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262225    5591 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:37:31.262796 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262228    5591 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:37:31.262796 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262230    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:37:31.262796 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262233    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:37:31.262796 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262236    5591 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:37:31.262796 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262243    5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:37:31.262796 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262246    5591 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:37:31.262796 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262249    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:37:31.262796 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262251    5591 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:37:31.262796 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262254    5591 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:37:31.262796 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262256    5591 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:37:31.262796 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262260    5591 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:37:31.262796 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262263    5591 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:37:31.262796 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262265    5591 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:37:31.262796 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262268    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:37:31.262796 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262271    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:37:31.262796 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262273    5591 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:37:31.262796 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262276    5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:37:31.263314 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262279    5591 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:37:31.263314 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262282    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:37:31.263314 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262284    5591 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:37:31.263314 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262287    5591 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:37:31.263314 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262290    5591 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:37:31.263314 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262293    5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:37:31.263314 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262295    5591 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:37:31.263314 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262299    5591 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:37:31.263314 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262301    5591 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:37:31.263314 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262304    5591 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:37:31.263314 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262307    5591 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:37:31.263314 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262309    5591 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:37:31.263314 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262312    5591 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:37:31.263314 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262316    5591 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:37:31.263314 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262319    5591 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:37:31.263314 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262321    5591 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:37:31.263314 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262324    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:37:31.263314 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262326    5591 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:37:31.263314 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262329    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:37:31.263314 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262331    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:37:31.263804 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262334    5591 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:37:31.263804 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262336    5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:37:31.263804 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262339    5591 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:37:31.263804 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262341    5591 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:37:31.263804 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262344    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:37:31.263804 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262347    5591 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:37:31.263804 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262349    5591 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:37:31.263804 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262353    5591 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:37:31.263804 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262357    5591 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:37:31.263804 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262360    5591 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:37:31.263804 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262363    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:37:31.263804 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262366    5591 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:37:31.263804 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262369    5591 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:37:31.263804 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262372    5591 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:37:31.263804 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262376    5591 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:37:31.263804 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262379    5591 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:37:31.263804 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262382    5591 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:37:31.263804 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262386    5591 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:37:31.263804 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262389    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:37:31.264297 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262392    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:37:31.264297 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262394    5591 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:37:31.264297 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262397    5591 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:37:31.264297 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262399    5591 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:37:31.264297 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262402    5591 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:37:31.264297 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262404    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:37:31.264297 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262407    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:37:31.264297 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262409    5591 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:37:31.264297 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262412    5591 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:37:31.264297 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262415    5591 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:37:31.264297 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262418    5591 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:37:31.264297 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262420    5591 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:37:31.264297 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262423    5591 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:37:31.264297 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262425    5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:37:31.264297 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262429    5591 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:37:31.264297 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262432    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:37:31.264297 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262434    5591 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:37:31.264297 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262437    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:37:31.264297 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262440    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:37:31.264297 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262443    5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:37:31.264858 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262445    5591 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:37:31.264858 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262448    5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:37:31.264858 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262450    5591 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:37:31.264858 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.262453    5591 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:37:31.264858 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263195    5591 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:37:31.264858 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263221    5591 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:37:31.264858 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263226    5591 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:37:31.264858 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263229    5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:37:31.264858 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263232    5591 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:37:31.264858 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263235    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:37:31.264858 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263237    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:37:31.264858 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263247    5591 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:37:31.264858 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263250    5591 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:37:31.264858 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263253    5591 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:37:31.264858 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263255    5591 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:37:31.264858 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263258    5591 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:37:31.264858 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263261    5591 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:37:31.264858 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263263    5591 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:37:31.264858 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263266    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:37:31.264858 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263269    5591 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:37:31.265420 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263271    5591 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:37:31.265420 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263274    5591 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:37:31.265420 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263276    5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:37:31.265420 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263279    5591 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:37:31.265420 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263281    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:37:31.265420 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263283    5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:37:31.265420 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263286    5591 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:37:31.265420 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263288    5591 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:37:31.265420 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263291    5591 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:37:31.265420 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263294    5591 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:37:31.265420 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263297    5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:37:31.265420 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263299    5591 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:37:31.265420 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263302    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:37:31.265420 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263304    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:37:31.265420 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263307    5591 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:37:31.265420 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263309    5591 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:37:31.265420 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263312    5591 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:37:31.265420 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263315    5591 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:37:31.265420 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263317    5591 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:37:31.265907 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263320    5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:37:31.265907 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263323    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:37:31.265907 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263326    5591 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:37:31.265907 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263328    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:37:31.265907 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263336    5591 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:37:31.265907 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263340    5591 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:37:31.265907 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263343    5591 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:37:31.265907 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263345    5591 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:37:31.265907 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263347    5591 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:37:31.265907 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263350    5591 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:37:31.265907 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263352    5591 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:37:31.265907 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263355    5591 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:37:31.265907 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263357    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:37:31.265907 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263359    5591 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:37:31.265907 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263362    5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:37:31.265907 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263364    5591 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:37:31.265907 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263367    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:37:31.265907 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263370    5591 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:37:31.265907 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263375    5591 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:37:31.266391 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263379    5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:37:31.266391 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263382    5591 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:37:31.266391 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263384    5591 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:37:31.266391 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263387    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:37:31.266391 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263391    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:37:31.266391 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263393    5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:37:31.266391 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263396    5591 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:37:31.266391 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263398    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:37:31.266391 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263401    5591 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:37:31.266391 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263404    5591 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:37:31.266391 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263407    5591 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:37:31.266391 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263409    5591 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:37:31.266391 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263411    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:37:31.266391 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263414    5591 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:37:31.266391 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263416    5591 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:37:31.266391 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263419    5591 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:37:31.266391 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263421    5591 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:37:31.266391 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263424    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:37:31.266391 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263426    5591 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:37:31.266391 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263429    5591 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:37:31.266883 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263432    5591 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:37:31.266883 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263434    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:37:31.266883 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263437    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:37:31.266883 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263439    5591 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:37:31.266883 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263441    5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:37:31.266883 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263444    5591 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:37:31.266883 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263446    5591 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:37:31.266883 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263449    5591 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:37:31.266883 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263452    5591 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:37:31.266883 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263454    5591 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:37:31.266883 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263457    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:37:31.266883 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263459    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:37:31.266883 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263462    5591 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:37:31.266883 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263465    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:37:31.266883 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263467    5591 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:37:31.266883 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263470    5591 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:37:31.266883 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263473    5591 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:37:31.266883 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263476    5591 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:37:31.266883 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263479    5591 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:37:31.266883 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263482    5591 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:37:31.267413 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263484    5591 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:37:31.267413 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263487    5591 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:37:31.267413 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263489    5591 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:37:31.267413 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263492    5591 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:37:31.267413 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263494    5591 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:37:31.267413 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263496    5591 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:37:31.267413 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263499    5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:37:31.267413 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263501    5591 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:37:31.267413 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263503    5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:37:31.267413 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263506    5591 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:37:31.267413 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263508    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:37:31.267413 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.263511    5591 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:37:31.267413 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264027    5591 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:37:31.267413 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264032    5591 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:37:31.267413 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264036    5591 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:37:31.267413 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264040    5591 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:37:31.267413 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264042    5591 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:37:31.267413 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264045    5591 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:37:31.267413 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264047    5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:37:31.267922 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264050    5591 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:37:31.267922 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264053    5591 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:37:31.267922 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264055    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:37:31.267922 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264058    5591 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:37:31.267922 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264061    5591 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:37:31.267922 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264064    5591 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:37:31.267922 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264066    5591 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:37:31.267922 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264069    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:37:31.267922 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264072    5591 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:37:31.267922 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264075    5591 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:37:31.267922 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264078    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:37:31.267922 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264080    5591 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:37:31.267922 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264083    5591 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:37:31.267922 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264085    5591 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:37:31.267922 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264088    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:37:31.267922 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264090    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:37:31.267922 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264094    5591 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:37:31.267922 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264098    5591 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:37:31.267922 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264101    5591 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:37:31.267922 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264104    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:37:31.268433 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264107    5591 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:37:31.268433 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264109    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:37:31.268433 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264112    5591 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:37:31.268433 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264115    5591 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:37:31.268433 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264117    5591 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:37:31.268433 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264120    5591 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:37:31.268433 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264123    5591 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:37:31.268433 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264125    5591 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:37:31.268433 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264128    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:37:31.268433 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264130    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:37:31.268433 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264133    5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:37:31.268433 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264135    5591 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:37:31.268433 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264138    5591 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:37:31.268433 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264140    5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:37:31.268433 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264142    5591 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:37:31.268433 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264145    5591 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:37:31.268433 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264147    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:37:31.268433 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264150    5591 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:37:31.268433 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264152    5591 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:37:31.268955 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264161    5591 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:37:31.268955 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264163    5591 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:37:31.268955 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264166    5591 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:37:31.268955 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264169    5591 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:37:31.268955 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264171    5591 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:37:31.268955 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264174    5591 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:37:31.268955 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264176    5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:37:31.268955 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264179    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:37:31.268955 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264181    5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:37:31.268955 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264184    5591 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:37:31.268955 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264186    5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:37:31.268955 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264189    5591 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:37:31.268955 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264191    5591 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:37:31.268955 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264194    5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:37:31.268955 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264198    5591 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:37:31.268955 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264220    5591 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:37:31.268955 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264225    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:37:31.268955 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264229    5591 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:37:31.268955 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264233    5591 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:37:31.269434 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264239    5591 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:37:31.269434 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264242    5591 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:37:31.269434 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264245    5591 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:37:31.269434 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264253    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:37:31.269434 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264256    5591 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:37:31.269434 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264258    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:37:31.269434 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264261    5591 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:37:31.269434 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264263    5591 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:37:31.269434 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264266    5591 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:37:31.269434 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264268    5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:37:31.269434 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264271    5591 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:37:31.269434 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264273    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:37:31.269434 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264276    5591 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:37:31.269434 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264280    5591 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:37:31.269434 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264283    5591 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:37:31.269434 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264285    5591 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:37:31.269434 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264288    5591 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:37:31.269434 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264290    5591 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:37:31.269434 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264292    5591 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:37:31.269901 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264295    5591 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:37:31.269901 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264297    5591 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:37:31.269901 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264300    5591 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:37:31.269901 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264304    5591 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:37:31.269901 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264306    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:37:31.269901 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264309    5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:37:31.269901 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264312    5591 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:37:31.269901 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264314    5591 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:37:31.269901 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264317    5591 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:37:31.269901 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264320    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:37:31.269901 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264322    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:37:31.269901 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264325    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:37:31.269901 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264327    5591 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:37:31.269901 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264330    5591 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:37:31.269901 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264338    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:37:31.269901 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264340    5591 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:37:31.269901 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264343    5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:37:31.269901 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264345    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:37:31.269901 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264348    5591 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:37:31.269901 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264351    5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:37:31.270451 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264353    5591 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:37:31.270451 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.264355    5591 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:37:31.270451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264449    5591 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 19:37:31.270451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264456    5591 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 19:37:31.270451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264469    5591 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 19:37:31.270451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264473    5591 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 19:37:31.270451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264480    5591 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 19:37:31.270451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264483    5591 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 19:37:31.270451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264488    5591 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 19:37:31.270451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264492    5591 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 19:37:31.270451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264495    5591 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 19:37:31.270451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264497    5591 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 19:37:31.270451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264501    5591 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 19:37:31.270451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264504    5591 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 19:37:31.270451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264507    5591 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 19:37:31.270451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264510    5591 flags.go:64] FLAG: --cgroup-root=""
Nov 03 19:37:31.270451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264513    5591 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 19:37:31.270451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264516    5591 flags.go:64] FLAG: --client-ca-file=""
Nov 03 19:37:31.270451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264519    5591 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 19:37:31.270451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264522    5591 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 19:37:31.270451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264528    5591 flags.go:64] FLAG: --cluster-domain=""
Nov 03 19:37:31.270451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264531    5591 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 19:37:31.270451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264534    5591 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 19:37:31.270451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264537    5591 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 19:37:31.271035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264541    5591 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 19:37:31.271035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264545    5591 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 19:37:31.271035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264548    5591 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 19:37:31.271035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264556    5591 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 19:37:31.271035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264560    5591 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 19:37:31.271035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264563    5591 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 19:37:31.271035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264566    5591 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 19:37:31.271035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264569    5591 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 19:37:31.271035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264572    5591 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 19:37:31.271035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264574    5591 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 19:37:31.271035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264579    5591 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 19:37:31.271035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264582    5591 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 19:37:31.271035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264584    5591 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 19:37:31.271035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264587    5591 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 19:37:31.271035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264592    5591 flags.go:64] FLAG: --enable-server="true"
Nov 03 19:37:31.271035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264595    5591 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 19:37:31.271035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264603    5591 flags.go:64] FLAG: --event-burst="100"
Nov 03 19:37:31.271035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264606    5591 flags.go:64] FLAG: --event-qps="50"
Nov 03 19:37:31.271035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264609    5591 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 19:37:31.271035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264612    5591 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 19:37:31.271035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264615    5591 flags.go:64] FLAG: --eviction-hard=""
Nov 03 19:37:31.271035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264620    5591 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 19:37:31.271035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264623    5591 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 19:37:31.271035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264626    5591 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 19:37:31.271035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264629    5591 flags.go:64] FLAG: --eviction-soft=""
Nov 03 19:37:31.271652 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264632    5591 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 19:37:31.271652 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264635    5591 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 19:37:31.271652 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264638    5591 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 19:37:31.271652 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264641    5591 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 19:37:31.271652 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264644    5591 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 19:37:31.271652 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264647    5591 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 19:37:31.271652 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264650    5591 flags.go:64] FLAG: --feature-gates=""
Nov 03 19:37:31.271652 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264654    5591 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 19:37:31.271652 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264657    5591 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 19:37:31.271652 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264660    5591 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 19:37:31.271652 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264663    5591 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 19:37:31.271652 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264672    5591 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 19:37:31.271652 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264676    5591 flags.go:64] FLAG: --help="false"
Nov 03 19:37:31.271652 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264678    5591 flags.go:64] FLAG: --hostname-override="ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:37:31.271652 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264681    5591 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 19:37:31.271652 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264684    5591 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 19:37:31.271652 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264687    5591 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 19:37:31.271652 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264690    5591 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 19:37:31.271652 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264693    5591 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 19:37:31.271652 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264697    5591 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 19:37:31.271652 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264699    5591 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 19:37:31.271652 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264703    5591 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 19:37:31.271652 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264706    5591 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 19:37:31.272236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264709    5591 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 19:37:31.272236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264712    5591 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 19:37:31.272236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264715    5591 flags.go:64] FLAG: --kube-reserved=""
Nov 03 19:37:31.272236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264718    5591 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 19:37:31.272236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264721    5591 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 19:37:31.272236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264724    5591 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 19:37:31.272236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264727    5591 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 19:37:31.272236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264729    5591 flags.go:64] FLAG: --lock-file=""
Nov 03 19:37:31.272236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264732    5591 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 19:37:31.272236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264735    5591 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 19:37:31.272236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264738    5591 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 19:37:31.272236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264744    5591 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 19:37:31.272236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264746    5591 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 19:37:31.272236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264751    5591 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 19:37:31.272236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264754    5591 flags.go:64] FLAG: --logging-format="text"
Nov 03 19:37:31.272236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264756    5591 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 19:37:31.272236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264769    5591 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 19:37:31.272236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264772    5591 flags.go:64] FLAG: --manifest-url=""
Nov 03 19:37:31.272236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264775    5591 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 19:37:31.272236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264779    5591 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 19:37:31.272236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264782    5591 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 19:37:31.272236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264792    5591 flags.go:64] FLAG: --max-pods="110"
Nov 03 19:37:31.272236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264796    5591 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 19:37:31.272236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264798    5591 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 19:37:31.272236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264801    5591 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 19:37:31.272862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264804    5591 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 19:37:31.272862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264807    5591 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 19:37:31.272862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264810    5591 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 19:37:31.272862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264813    5591 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 19:37:31.272862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264820    5591 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 19:37:31.272862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264824    5591 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 19:37:31.272862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264827    5591 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 19:37:31.272862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264830    5591 flags.go:64] FLAG: --pod-cidr=""
Nov 03 19:37:31.272862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264833    5591 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 19:37:31.272862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264835    5591 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 19:37:31.272862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264839    5591 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 19:37:31.272862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264841    5591 flags.go:64] FLAG: --port="10250"
Nov 03 19:37:31.272862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264844    5591 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 19:37:31.272862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264847    5591 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0b788c8e870c8382f"
Nov 03 19:37:31.272862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264850    5591 flags.go:64] FLAG: --qos-reserved=""
Nov 03 19:37:31.272862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264853    5591 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 19:37:31.272862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264856    5591 flags.go:64] FLAG: --register-node="true"
Nov 03 19:37:31.272862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264859    5591 flags.go:64] FLAG: --register-with-taints=""
Nov 03 19:37:31.272862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264863    5591 flags.go:64] FLAG: --registry-burst="10"
Nov 03 19:37:31.272862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264866    5591 flags.go:64] FLAG: --registry-qps="5"
Nov 03 19:37:31.272862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264869    5591 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 19:37:31.272862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264873    5591 flags.go:64] FLAG: --reserved-memory=""
Nov 03 19:37:31.272862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264877    5591 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 19:37:31.272862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264880    5591 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 19:37:31.272862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264883    5591 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 19:37:31.272862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264886    5591 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 19:37:31.273597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264889    5591 flags.go:64] FLAG: --runonce="false"
Nov 03 19:37:31.273597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264891    5591 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 19:37:31.273597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264894    5591 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 19:37:31.273597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264898    5591 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 19:37:31.273597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264901    5591 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 19:37:31.273597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264904    5591 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 19:37:31.273597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264907    5591 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 19:37:31.273597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264910    5591 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 19:37:31.273597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264913    5591 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 19:37:31.273597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264915    5591 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 19:37:31.273597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264918    5591 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 19:37:31.273597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264922    5591 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 19:37:31.273597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264925    5591 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 19:37:31.273597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264928    5591 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 19:37:31.273597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264931    5591 flags.go:64] FLAG: --system-cgroups=""
Nov 03 19:37:31.273597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264934    5591 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 19:37:31.273597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264939    5591 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 19:37:31.273597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264942    5591 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 19:37:31.273597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264944    5591 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 19:37:31.273597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264951    5591 flags.go:64] FLAG: --tls-min-version=""
Nov 03 19:37:31.273597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264954    5591 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 19:37:31.273597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264957    5591 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 19:37:31.273597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264960    5591 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 19:37:31.273597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264963    5591 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 19:37:31.273597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264966    5591 flags.go:64] FLAG: --v="2"
Nov 03 19:37:31.274234 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264971    5591 flags.go:64] FLAG: --version="false"
Nov 03 19:37:31.274234 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264979    5591 flags.go:64] FLAG: --vmodule=""
Nov 03 19:37:31.274234 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264983    5591 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 19:37:31.274234 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.264986    5591 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 19:37:31.274234 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265126    5591 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:37:31.274234 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265130    5591 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:37:31.274234 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265133    5591 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:37:31.274234 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265135    5591 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:37:31.274234 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265138    5591 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:37:31.274234 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265141    5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:37:31.274234 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265143    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:37:31.274234 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265146    5591 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:37:31.274234 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265149    5591 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:37:31.274234 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265152    5591 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:37:31.274234 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265154    5591 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:37:31.274234 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265156    5591 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:37:31.274234 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265159    5591 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:37:31.274234 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265162    5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:37:31.274234 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265165    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:37:31.274234 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265168    5591 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:37:31.274234 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265171    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:37:31.274818 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265173    5591 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:37:31.274818 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265176    5591 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:37:31.274818 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265178    5591 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:37:31.274818 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265181    5591 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:37:31.274818 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265183    5591 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:37:31.274818 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265186    5591 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:37:31.274818 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265188    5591 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:37:31.274818 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265191    5591 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:37:31.274818 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265194    5591 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:37:31.274818 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265196    5591 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:37:31.274818 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265216    5591 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:37:31.274818 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265220    5591 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:37:31.274818 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265224    5591 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:37:31.274818 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265227    5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:37:31.274818 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265230    5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:37:31.274818 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265233    5591 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:37:31.274818 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265235    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:37:31.274818 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265238    5591 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:37:31.274818 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265240    5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:37:31.274818 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265243    5591 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:37:31.275663 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265245    5591 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:37:31.275663 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265248    5591 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:37:31.275663 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265250    5591 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:37:31.275663 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265254    5591 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:37:31.275663 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265256    5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:37:31.275663 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265259    5591 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:37:31.275663 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265262    5591 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:37:31.275663 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265264    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:37:31.275663 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265267    5591 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:37:31.275663 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265270    5591 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:37:31.275663 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265273    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:37:31.275663 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265275    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:37:31.275663 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265278    5591 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:37:31.275663 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265280    5591 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:37:31.275663 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265283    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:37:31.275663 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265286    5591 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:37:31.275663 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265288    5591 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:37:31.275663 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265291    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:37:31.275663 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265293    5591 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:37:31.275663 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265296    5591 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:37:31.276393 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265298    5591 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:37:31.276393 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265301    5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:37:31.276393 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265303    5591 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:37:31.276393 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265306    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:37:31.276393 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265310    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:37:31.276393 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265313    5591 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:37:31.276393 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265315    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:37:31.276393 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265317    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:37:31.276393 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265320    5591 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:37:31.276393 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265323    5591 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:37:31.276393 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265325    5591 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:37:31.276393 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265327    5591 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:37:31.276393 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265330    5591 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:37:31.276393 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265332    5591 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:37:31.276393 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265335    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:37:31.276393 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265337    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:37:31.276393 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265340    5591 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:37:31.276393 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265342    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:37:31.276393 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265345    5591 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:37:31.276959 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265347    5591 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:37:31.276959 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265350    5591 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:37:31.276959 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265353    5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:37:31.276959 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265355    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:37:31.276959 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265360    5591 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:37:31.276959 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265364    5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:37:31.276959 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265367    5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:37:31.276959 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265369    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:37:31.276959 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265372    5591 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:37:31.276959 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265375    5591 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:37:31.276959 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265377    5591 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:37:31.276959 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265380    5591 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:37:31.276959 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265382    5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:37:31.276959 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265385    5591 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:37:31.276959 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265387    5591 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:37:31.276959 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265390    5591 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:37:31.276959 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265392    5591 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:37:31.276959 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265397    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:37:31.276959 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265399    5591 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:37:31.277545 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265402    5591 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:37:31.277545 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265404    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:37:31.277545 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265407    5591 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:37:31.277545 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265409    5591 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:37:31.277545 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265411    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:37:31.277545 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265415    5591 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:37:31.277545 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265420    5591 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:37:31.277545 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265423    5591 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:37:31.277545 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265426    5591 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:37:31.277545 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265429    5591 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:37:31.277545 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265431    5591 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:37:31.277545 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.265459    5591 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:37:31.277545 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265495    5591 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:37:31.277545 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265500    5591 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:37:31.277545 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265503    5591 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:37:31.278009 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265506    5591 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:37:31.278009 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265510    5591 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:37:31.278009 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265512    5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:37:31.278009 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265515    5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:37:31.278009 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265517    5591 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:37:31.278009 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265520    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:37:31.278009 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265522    5591 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:37:31.278009 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265525    5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:37:31.278009 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265527    5591 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:37:31.278009 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265529    5591 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:37:31.278009 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265532    5591 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:37:31.278009 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265534    5591 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:37:31.278009 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265537    5591 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:37:31.278009 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265539    5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:37:31.278009 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265542    5591 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:37:31.278009 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265546    5591 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:37:31.278009 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265548    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:37:31.278009 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265551    5591 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:37:31.278009 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265553    5591 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:37:31.278009 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265556    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:37:31.278917 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265558    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:37:31.278917 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265560    5591 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:37:31.278917 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265563    5591 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:37:31.278917 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265565    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:37:31.278917 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265567    5591 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:37:31.278917 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265570    5591 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:37:31.278917 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265572    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:37:31.278917 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265575    5591 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:37:31.278917 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265577    5591 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:37:31.278917 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265580    5591 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:37:31.278917 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265582    5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:37:31.278917 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265585    5591 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:37:31.278917 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265587    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:37:31.278917 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265590    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:37:31.278917 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265593    5591 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:37:31.278917 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265595    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:37:31.278917 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265597    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:37:31.278917 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265600    5591 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:37:31.278917 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265602    5591 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:37:31.278917 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265605    5591 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:37:31.279788 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265607    5591 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:37:31.279788 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265610    5591 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:37:31.279788 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265612    5591 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:37:31.279788 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265615    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:37:31.279788 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265617    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:37:31.279788 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265619    5591 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:37:31.279788 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265622    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:37:31.279788 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265625    5591 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:37:31.279788 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265628    5591 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:37:31.279788 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265630    5591 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:37:31.279788 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265632    5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:37:31.279788 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265635    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:37:31.279788 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265638    5591 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:37:31.279788 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265642    5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:37:31.279788 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265644    5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:37:31.279788 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265647    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:37:31.279788 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265649    5591 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:37:31.279788 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265651    5591 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:37:31.279788 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265654    5591 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:37:31.280342 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265656    5591 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:37:31.280342 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265659    5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:37:31.280342 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265661    5591 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:37:31.280342 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265664    5591 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:37:31.280342 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265667    5591 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:37:31.280342 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265669    5591 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:37:31.280342 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265672    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:37:31.280342 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265674    5591 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:37:31.280342 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265677    5591 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:37:31.280342 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265679    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:37:31.280342 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265681    5591 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:37:31.280342 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265684    5591 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:37:31.280342 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265686    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:37:31.280342 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265690    5591 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:37:31.280342 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265695    5591 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:37:31.280342 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265698    5591 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:37:31.280342 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265701    5591 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:37:31.280342 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265703    5591 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:37:31.280342 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265706    5591 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:37:31.280929 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265708    5591 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:37:31.280929 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265712    5591 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:37:31.280929 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265714    5591 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:37:31.280929 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265717    5591 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:37:31.280929 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265719    5591 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:37:31.280929 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265722    5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:37:31.280929 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265725    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:37:31.280929 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265728    5591 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:37:31.280929 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265730    5591 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:37:31.280929 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265732    5591 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:37:31.280929 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265735    5591 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:37:31.280929 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265737    5591 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:37:31.280929 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265739    5591 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:37:31.280929 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265742    5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:37:31.280929 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265744    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:37:31.280929 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265747    5591 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:37:31.280929 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265749    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:37:31.280929 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265752    5591 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:37:31.280929 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265754    5591 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:37:31.280929 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265758    5591 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:37:31.280929 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265761    5591 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:37:31.281603 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265764    5591 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:37:31.281603 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265767    5591 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:37:31.281603 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265769    5591 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:37:31.281603 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.265771    5591 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:37:31.281603 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.269847    5591 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 19:37:31.281603 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.269863    5591 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 19:37:31.281603 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.269881    5591 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 19:37:31.281603 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.269886    5591 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 19:37:31.281603 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.269955    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:37:31.281603 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.269960    5591 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:37:31.281603 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.269964    5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:37:31.281603 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.269966    5591 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:37:31.281603 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.269969    5591 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:37:31.281603 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.269972    5591 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:37:31.281603 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.269974    5591 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:37:31.281603 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.269977    5591 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:37:31.281603 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.269980    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:37:31.281603 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.269982    5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:37:31.281603 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.269985    5591 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:37:31.281603 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.269987    5591 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:37:31.282089 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.269990    5591 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:37:31.282089 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.269992    5591 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:37:31.282089 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.269995    5591 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:37:31.282089 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.269998    5591 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:37:31.282089 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270000    5591 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:37:31.282089 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270003    5591 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:37:31.282089 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270005    5591 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:37:31.282089 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270008    5591 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:37:31.282089 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270016    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:37:31.282089 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270019    5591 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:37:31.282089 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270022    5591 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:37:31.282089 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270024    5591 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:37:31.282089 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270027    5591 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:37:31.282089 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270029    5591 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:37:31.282089 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270032    5591 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:37:31.282089 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270037    5591 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:37:31.282089 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270041    5591 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:37:31.282089 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270045    5591 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:37:31.282089 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270047    5591 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:37:31.282089 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270051    5591 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:37:31.282628 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270054    5591 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:37:31.282628 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270057    5591 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:37:31.282628 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270059    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:37:31.282628 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270062    5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:37:31.282628 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270064    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:37:31.282628 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270067    5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:37:31.282628 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270069    5591 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:37:31.282628 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270072    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:37:31.282628 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270074    5591 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:37:31.282628 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270076    5591 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:37:31.282628 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270079    5591 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:37:31.282628 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270081    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:37:31.282628 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270084    5591 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:37:31.282628 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270086    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:37:31.282628 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270089    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:37:31.282628 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270091    5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:37:31.282628 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270094    5591 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:37:31.282628 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270096    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:37:31.282628 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270099    5591 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:37:31.282628 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270102    5591 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:37:31.283114 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270104    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:37:31.283114 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270112    5591 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:37:31.283114 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270115    5591 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:37:31.283114 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270117    5591 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:37:31.283114 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270120    5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:37:31.283114 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270122    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:37:31.283114 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270125    5591 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:37:31.283114 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270128    5591 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:37:31.283114 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270130    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:37:31.283114 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270133    5591 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:37:31.283114 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270135    5591 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:37:31.283114 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270138    5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:37:31.283114 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270140    5591 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:37:31.283114 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270142    5591 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:37:31.283114 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270145    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:37:31.283114 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270148    5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:37:31.283114 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270151    5591 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:37:31.283114 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270155    5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:37:31.283114 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270158    5591 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:37:31.283599 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270160    5591 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:37:31.283599 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270163    5591 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:37:31.283599 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270165    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:37:31.283599 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270168    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:37:31.283599 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270170    5591 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:37:31.283599 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270173    5591 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:37:31.283599 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270176    5591 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:37:31.283599 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270178    5591 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:37:31.283599 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270180    5591 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:37:31.283599 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270183    5591 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:37:31.283599 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270185    5591 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:37:31.283599 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270188    5591 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:37:31.283599 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270190    5591 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:37:31.283599 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270218    5591 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:37:31.283599 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270223    5591 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:37:31.283599 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270233    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:37:31.283599 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270236    5591 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:37:31.283599 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270238    5591 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:37:31.283599 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270241    5591 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:37:31.283599 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270243    5591 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:37:31.284083 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270246    5591 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:37:31.284083 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270248    5591 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:37:31.284083 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270251    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:37:31.284083 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270253    5591 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:37:31.284083 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270256    5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:37:31.284083 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270258    5591 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:37:31.284083 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270261    5591 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:37:31.284083 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270263    5591 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:37:31.284083 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270265    5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:37:31.284083 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270268    5591 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:37:31.284083 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270271    5591 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:37:31.284083 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270273    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:37:31.284083 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270276    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:37:31.284083 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270278    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:37:31.284083 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270281    5591 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:37:31.284464 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.270308    5591 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:37:31.284464 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270507    5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:37:31.284464 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270513    5591 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:37:31.284464 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270516    5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:37:31.284464 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270518    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:37:31.284464 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270521    5591 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:37:31.284464 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270523    5591 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:37:31.284464 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270526    5591 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:37:31.284464 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270528    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:37:31.284464 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270531    5591 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:37:31.284464 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270533    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:37:31.284464 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270535    5591 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:37:31.284464 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270538    5591 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:37:31.284464 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270546    5591 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:37:31.284464 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270548    5591 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:37:31.284838 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270551    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:37:31.284838 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270553    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:37:31.284838 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270557    5591 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:37:31.284838 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270559    5591 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:37:31.284838 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270561    5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:37:31.284838 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270564    5591 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:37:31.284838 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270566    5591 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:37:31.284838 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270569    5591 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:37:31.284838 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270571    5591 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:37:31.284838 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270573    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:37:31.284838 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270576    5591 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:37:31.284838 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270578    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:37:31.284838 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270581    5591 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:37:31.284838 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270583    5591 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:37:31.284838 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270585    5591 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:37:31.284838 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270588    5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:37:31.284838 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270590    5591 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:37:31.284838 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270593    5591 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:37:31.284838 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270595    5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:37:31.284838 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270597    5591 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:37:31.285404 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270600    5591 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:37:31.285404 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270602    5591 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:37:31.285404 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270604    5591 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:37:31.285404 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270607    5591 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:37:31.285404 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270609    5591 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:37:31.285404 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270611    5591 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:37:31.285404 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270614    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:37:31.285404 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270616    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:37:31.285404 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270619    5591 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:37:31.285404 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270621    5591 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:37:31.285404 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270624    5591 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:37:31.285404 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270631    5591 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:37:31.285404 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270634    5591 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:37:31.285404 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270637    5591 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:37:31.285404 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270639    5591 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:37:31.285404 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270642    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:37:31.285404 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270644    5591 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:37:31.285404 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270647    5591 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:37:31.285404 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270649    5591 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:37:31.285404 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270652    5591 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:37:31.285891 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270654    5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:37:31.285891 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270657    5591 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:37:31.285891 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270659    5591 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:37:31.285891 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270661    5591 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:37:31.285891 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270664    5591 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:37:31.285891 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270666    5591 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:37:31.285891 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270669    5591 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:37:31.285891 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270671    5591 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:37:31.285891 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270674    5591 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:37:31.285891 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270678    5591 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:37:31.285891 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270682    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:37:31.285891 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270685    5591 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:37:31.285891 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270688    5591 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:37:31.285891 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270691    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:37:31.285891 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270693    5591 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:37:31.285891 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270696    5591 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:37:31.285891 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270701    5591 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:37:31.285891 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270703    5591 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:37:31.285891 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270706    5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:37:31.286372 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270708    5591 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:37:31.286372 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270710    5591 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:37:31.286372 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270713    5591 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:37:31.286372 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270715    5591 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:37:31.286372 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270718    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:37:31.286372 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270726    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:37:31.286372 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270729    5591 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:37:31.286372 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270732    5591 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:37:31.286372 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270734    5591 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:37:31.286372 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270737    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:37:31.286372 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270739    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:37:31.286372 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270741    5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:37:31.286372 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270744    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:37:31.286372 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270747    5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:37:31.286372 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270749    5591 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:37:31.286372 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270752    5591 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:37:31.286372 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270754    5591 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:37:31.286372 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270756    5591 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:37:31.286372 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270759    5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:37:31.286372 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270761    5591 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:37:31.286849 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270764    5591 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:37:31.286849 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270766    5591 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:37:31.286849 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270768    5591 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:37:31.286849 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270771    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:37:31.286849 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270773    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:37:31.286849 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270777    5591 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:37:31.286849 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270780    5591 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:37:31.286849 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270783    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:37:31.286849 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270785    5591 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:37:31.286849 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270789    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:37:31.286849 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270792    5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:37:31.286849 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270794    5591 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:37:31.286849 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270797    5591 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:37:31.286849 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.270820    5591 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 19:37:31.286849 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270864    5591 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 19:37:31.286849 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270869    5591 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 19:37:31.287243 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270872    5591 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 19:37:31.287243 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270875    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 19:37:31.287243 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270886    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 19:37:31.287243 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270888    5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 19:37:31.287243 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270891    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 19:37:31.287243 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270894    5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 19:37:31.287243 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270896    5591 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 19:37:31.287243 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270899    5591 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 19:37:31.287243 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270901    5591 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 19:37:31.287243 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270903    5591 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 19:37:31.287243 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270906    5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 19:37:31.287243 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270908    5591 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 19:37:31.287243 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270911    5591 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 19:37:31.287243 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270913    5591 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 19:37:31.287243 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270916    5591 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 19:37:31.287243 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270918    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 19:37:31.287243 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270921    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 19:37:31.287243 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270924    5591 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 19:37:31.287243 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270927    5591 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 19:37:31.287694 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270930    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 19:37:31.287694 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270932    5591 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 19:37:31.287694 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270935    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 19:37:31.287694 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270937    5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 19:37:31.287694 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270940    5591 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 19:37:31.287694 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270942    5591 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 19:37:31.287694 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270944    5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 19:37:31.287694 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270949    5591 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 19:37:31.287694 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270951    5591 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 19:37:31.287694 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270953    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 19:37:31.287694 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270956    5591 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 19:37:31.287694 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270958    5591 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 19:37:31.287694 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270961    5591 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 19:37:31.287694 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270964    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 19:37:31.287694 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270967    5591 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 19:37:31.287694 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270969    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 19:37:31.287694 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270977    5591 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 19:37:31.287694 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270980    5591 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 19:37:31.287694 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270982    5591 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 19:37:31.288181 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270985    5591 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 19:37:31.288181 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270987    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 19:37:31.288181 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270990    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 19:37:31.288181 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270992    5591 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 19:37:31.288181 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270995    5591 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 19:37:31.288181 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.270997    5591 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 19:37:31.288181 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271000    5591 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 19:37:31.288181 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271002    5591 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 19:37:31.288181 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271005    5591 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 19:37:31.288181 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271007    5591 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 19:37:31.288181 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271010    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 19:37:31.288181 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271012    5591 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 19:37:31.288181 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271014    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 19:37:31.288181 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271017    5591 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 19:37:31.288181 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271019    5591 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 19:37:31.288181 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271021    5591 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 19:37:31.288181 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271024    5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 19:37:31.288181 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271026    5591 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 19:37:31.288181 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271029    5591 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 19:37:31.288181 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271031    5591 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 19:37:31.288676 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271035    5591 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 19:37:31.288676 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271037    5591 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 19:37:31.288676 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271040    5591 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 19:37:31.288676 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271042    5591 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 19:37:31.288676 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271045    5591 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 19:37:31.288676 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271047    5591 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 19:37:31.288676 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271050    5591 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 19:37:31.288676 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271053    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 19:37:31.288676 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271056    5591 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 19:37:31.288676 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271058    5591 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 19:37:31.288676 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271066    5591 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 19:37:31.288676 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271069    5591 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 19:37:31.288676 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271072    5591 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 19:37:31.288676 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271074    5591 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 19:37:31.288676 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271076    5591 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 19:37:31.288676 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271079    5591 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 19:37:31.288676 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271081    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 19:37:31.288676 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271083    5591 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 19:37:31.288676 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271086    5591 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 19:37:31.288676 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271088    5591 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 19:37:31.289171 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271091    5591 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 19:37:31.289171 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271093    5591 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 19:37:31.289171 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271096    5591 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 19:37:31.289171 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271098    5591 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 19:37:31.289171 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271100    5591 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 19:37:31.289171 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271103    5591 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 19:37:31.289171 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271105    5591 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 19:37:31.289171 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271108    5591 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 19:37:31.289171 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271110    5591 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 19:37:31.289171 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271113    5591 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 19:37:31.289171 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271117    5591 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 19:37:31.289171 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271120    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 19:37:31.289171 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271123    5591 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 19:37:31.289171 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271126    5591 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 19:37:31.289171 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271129    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 19:37:31.289171 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271138    5591 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 19:37:31.289171 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271141    5591 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 19:37:31.289171 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271144    5591 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 19:37:31.289171 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271147    5591 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 19:37:31.289635 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271149    5591 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 19:37:31.289635 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271152    5591 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 19:37:31.289635 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271155    5591 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 19:37:31.289635 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271157    5591 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 19:37:31.289635 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271165    5591 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 19:37:31.289635 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271168    5591 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 19:37:31.289635 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:31.271171    5591 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 19:37:31.289635 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.271246    5591 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 19:37:31.289635 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.272272    5591 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 19:37:31.289635 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.272327    5591 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 19:37:31.289635 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.272514    5591 server.go:1014] "Starting client certificate rotation"
Nov 03 19:37:31.289635 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.272661    5591 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 19:37:31.289635 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.272783    5591 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:31:19 +0000 UTC" deadline="2028-06-21 01:20:37.787143164 +0000 UTC"
Nov 03 19:37:31.289635 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.272841    5591 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14285h43m6.51430398s"
Nov 03 19:37:31.289635 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.273094    5591 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:37:31.289635 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.274423    5591 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 19:37:31.290072 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.277415    5591 log.go:25] "Validated CRI v1 runtime API"
Nov 03 19:37:31.290072 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.279870    5591 log.go:25] "Validated CRI v1 image API"
Nov 03 19:37:31.290072 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.280676    5591 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 19:37:31.290072 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.282773    5591 fs.go:135] Filesystem UUIDs: map[4b595758-3b71-4bf0-b413-11dd05eb24ce:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e4defb7b-9f8b-4203-b68e-c808c060f10e:/dev/nvme0n1p3]
Nov 03 19:37:31.290197 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.282794    5591 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0eaf6115f4579620b0f14c3598a9d52a4cd6355129bde16bac809b7d568be2ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0eaf6115f4579620b0f14c3598a9d52a4cd6355129bde16bac809b7d568be2ec/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0f241ca48f494c140d074ff6fcfb46095a2ca4874c8c33c1d31069343b99f3dd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0f241ca48f494c140d074ff6fcfb46095a2ca4874c8c33c1d31069343b99f3dd/userdata/shm major:0 minor:351 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/297782fc2eb6ce8a7dd368a98ac3593c216345a3f5ff82942c8d18edba787c48/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/297782fc2eb6ce8a7dd368a98ac3593c216345a3f5ff82942c8d18edba787c48/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e6bc54351c35f29c80caf98aa51d0e66f770cc73a6380359128f72ab58b65d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e6bc54351c35f29c80caf98aa51d0e66f770cc73a6380359128f72ab58b65d8/userdata/shm major:0 minor:410 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48770e51f689ef669cbfb99db1b67bb09f19a6ca313a5d7c90707b36979f2dc2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48770e51f689ef669cbfb99db1b67bb09f19a6ca313a5d7c90707b36979f2dc2/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5672d069f537ac87a1d7070bb4a617199f7a10c736d6206a803d2a3273952c9a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5672d069f537ac87a1d7070bb4a617199f7a10c736d6206a803d2a3273952c9a/userdata/shm major:0 minor:274 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57562e83005ea47370f2c057b8116aa6f5e7fd92322963cdc16c9849d4ca87cd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57562e83005ea47370f2c057b8116aa6f5e7fd92322963cdc16c9849d4ca87cd/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a7f41c5228d6776608f44fd25798b14838c46bc7bdb1a5f798a7071d15e8c35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a7f41c5228d6776608f44fd25798b14838c46bc7bdb1a5f798a7071d15e8c35/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62f8eb6fd8dcd0e75732103842fb7277ccb2358ace74075dc8fa03dc3a13af04/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62f8eb6fd8dcd0e75732103842fb7277ccb2358ace74075dc8fa03dc3a13af04/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/77de5d1cd3a6ba081b6080676f08c8275491705c308f21e922697ae3ed8f06dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/77de5d1cd3a6ba081b6080676f08c8275491705c308f21e922697ae3ed8f06dc/userdata/shm major:0 minor:305 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c27ff198d55b17c368881572802b452da4123f9e871a6827814ff5a986e86b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c27ff198d55b17c368881572802b452da4123f9e871a6827814ff5a986e86b8/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8edd2f162297442f9f77e5ba98660f4c491934551e31553c054c9eecdd98d5b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8edd2f162297442f9f77e5ba98660f4c491934551e31553c054c9eecdd98d5b1/userdata/shm major:0 minor:97 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91ac845a4e0fd4d48706806d9296092656f225636a066a05db0d42d15dee5ec3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91ac845a4e0fd4d48706806d9296092656f225636a066a05db0d42d15dee5ec3/userdata/shm major:0 minor:414 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c3e81c5328eec5ff651a59728145ec14d13cc2e0bd31f41d97f17321fae9a3e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c3e81c5328eec5ff651a59728145ec14d13cc2e0bd31f41d97f17321fae9a3e3/userdata/shm major:0 minor:276 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca5a374ce20ae82ec77cdcd80570f06fe84262dbb9729cd8a71a5c77f1bf9373/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca5a374ce20ae82ec77cdcd80570f06fe84262dbb9729cd8a71a5c77f1bf9373/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d8fa1f9d28b5fd9c66cab82f903427748428dc067259baa487701b7c3e950780/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d8fa1f9d28b5fd9c66cab82f903427748428dc067259baa487701b7c3e950780/userdata/shm major:0 minor:412 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da480f56accb6891c3c0d00769c98771f2bd08c62e5ec777516d31555cb5a27d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da480f56accb6891c3c0d00769c98771f2bd08c62e5ec777516d31555cb5a27d/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dc002ae6adfc9db21f86304ef7cef337adb0de17d39be256b6634f686cf10d88/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dc002ae6adfc9db21f86304ef7cef337adb0de17d39be256b6634f686cf10d88/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3b53f7107aff24d23d341b7d5189301d877d896ed6e3fd7f52eed5a7b978636/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3b53f7107aff24d23d341b7d5189301d877d896ed6e3fd7f52eed5a7b978636/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e4d01dcd5adba2d4060227e841b164fc8265e16eec03d9d866e829e9e8e8af84/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e4d01dcd5adba2d4060227e841b164fc8265e16eec03d9d866e829e9e8e8af84/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f11f7ea9334844e8aadb4a2b85717570432be27063013dd93e6846a27bc4bf14/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f11f7ea9334844e8aadb4a2b85717570432be27063013dd93e6846a27bc4bf14/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f45ff49a9cf6737e1a1324c65dd202171c2106d3b3d10b63706762b06b9b7cf4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f45ff49a9cf6737e1a1324c65dd202171c2106d3b3d10b63706762b06b9b7cf4/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/055215b4-9de9-494c-bbff-288cc97baf2f/volumes/kubernetes.io~projected/kube-api-access-66pfk:{mountpoint:/var/lib/kubelet/pods/055215b4-9de9-494c-bbff-288cc97baf2f/volumes/kubernetes.io~projected/kube-api-access-66pfk major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1cd0db81-8027-4c6a-b33c-3b4545351eac/volumes/kubernetes.io~projected/kube-api-access-gvddx:{mountpoint:/var/lib/kubelet/pods/1cd0db81-8027-4c6a-b33c-3b4545351eac/volumes/kubernetes.io~projected/kube-api-access-gvddx major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1cd0db81-8027-4c6a-b33c-3b4545351eac/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/1cd0db81-8027-4c6a-b33c-3b4545351eac/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1cd0db81-8027-4c6a-b33c-3b4545351eac/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/1cd0db81-8027-4c6a-b33c-3b4545351eac/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/300a6502-6932-49fe-936a-ba0c16978c94/volumes/kubernetes.io~projected/kube-api-access-x8hqp:{mountpoint:/var/lib/kubelet/pods/300a6502-6932-49fe-936a-ba0c16978c94/volumes/kubernetes.io~projected/kube-api-access-x8hqp major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/300a6502-6932-49fe-936a-ba0c16978c94/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/300a6502-6932-49fe-936a-ba0c16978c94/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/300a6502-6932-49fe-936a-ba0c16978c94/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/300a6502-6932-49fe-936a-ba0c16978c94/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32452ae6-08d2-4fb2-863d-72dc09d4c0a2/volumes/kubernetes.io~projected/kube-api-access-dgxw5:{mountpoint:/var/lib/kubelet/pods/32452ae6-08d2-4fb2-863d-72dc09d4c0a2/volumes/kubernetes.io~projected/kube-api-access-dgxw5 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ac8ac4b-fef2-475d-a40b-9db77cded4d3/volumes/kubernetes.io~projected/kube-api-access-jp6g4:{mountpoint:/var/lib/kubelet/pods/3ac8ac4b-fef2-475d-a40b-9db77cded4d3/volumes/kubernetes.io~projected/kube-api-access-jp6g4 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ac8ac4b-fef2-475d-a40b-9db77cded4d3/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/3ac8ac4b-fef2-475d-a40b-9db77cded4d3/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/630922bd-4efc-4b44-96d1-fa55bce60e21/volumes/kubernetes.io~projected/kube-api-access-jktld:{mountpoint:/var/lib/kubelet/pods/630922bd-4efc-4b44-96d1-fa55bce60e21/volumes/kubernetes.io~projected/kube-api-access-jktld major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65b590ed-e1a5-467f-9283-7c875dd9ee24/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/65b590ed-e1a5-467f-9283-7c875dd9ee24/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65b590ed-e1a5-467f-9283-7c875dd9ee24/volumes/kubernetes.io~projected/kube-api-access-vgdxj:{mountpoint:/var/lib/kubelet/pods/65b590ed-e1a5-467f-9283-7c875dd9ee24/volumes/kubernetes.io~projected/kube-api-access-vgdxj major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65b590ed-e1a5-467f-9283-7c875dd9ee24/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/65b590ed-e1a5-467f-9283-7c875dd9ee24/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71c679be-e63f-4ad4-a1b2-5474ec4fa125/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/71c679be-e63f-4ad4-a1b2-5474ec4fa125/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71c679be-e63f-4ad4-a1b2-5474ec4fa125/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/71c679be-e63f-4ad4-a1b2-5474ec4fa125/volumes/kubernetes.io~empty-dir/tmp major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71c679be-e63f-4ad4-a1b2-5474ec4fa125/volumes/kubernetes.io~projected/kube-api-access-4zmfm:{mountpoint:/var/lib/kubelet/pods/71c679be-e63f-4ad4-a1b2-5474ec4fa125/volumes/kubernetes.io~projected/kube-api-access-4zmfm major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/803910f1-a1f3-4316-ada9-1890e46bbda1/volumes/kubernetes.io~projected/kube-api-access-q6rwg:{mountpoint:/var/lib/kubelet/pods/803910f1-a1f3-4316-ada9-1890e46bbda1/volumes/kubernetes.io~projected/kube-api-access-q6rwg major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/803910f1-a1f3-4316-ada9-1890e46bbda1/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/803910f1-a1f3-4316-ada9-1890e46bbda1/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/803910f1-a1f3-4316-ada9-1890e46bbda1/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/803910f1-a1f3-4316-ada9-1890e46bbda1/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8771d72e-12a3-429f-b458-e22298238393/volumes/kubernetes.io~projected/kube-api-access-5sgbt:{mountpoint:/var/lib/kubelet/pods/8771d72e-12a3-429f-b458-e22298238393/volumes/kubernetes.io~projected/kube-api-access-5sgbt major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8771d72e-12a3-429f-b458-e22298238393/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/8771d72e-12a3-429f-b458-e22298238393/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88a27c5e-de31-4090-be4d-6b8d6551e4c7/volumes/kubernetes.io~projected/kube-api-access-5nx7w:{mountpoint:/var/lib/kubelet/pods/88a27c5e-de31-4090-be4d-6b8d6551e4c7/volumes/kubernetes.io~projected/kube-api-access-5nx7w major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volume-subpaths/web-config/alertmanager/9 major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volume-subpaths/web-config/config-reloader/6 major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~empty-dir/config-out major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~projected/kube-api-access-tmggx:{mountpoint:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~projected/kube-api-access-tmggx major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~projected/tls-assets major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~secret/config-volume major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~secret/web-config major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98d32fbe-5011-47a8-b937-725a8cad95a4/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/98d32fbe-5011-47a8-b937-725a8cad95a4/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:408 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1cda986-f386-462d-9956-acc2a4de6e32/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b1cda986-f386-462d-9956-acc2a4de6e32/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4a42db6-0377-4a5a-94ab-be3f82b332aa/volumes/kubernetes.io~projected/kube-api-access-tzmv5:{mountpoint:/var/lib/kubelet/pods/b4a42db6-0377-4a5a-94ab-be3f82b332aa/volumes/kubernetes.io~projected/kube-api-access-tzmv5 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7e50fe5-0dd0-4f59-bc5f-99199500790b/volumes/kubernetes.io~projected/kube-api-access-7n2qg:{mountpoint:/var/lib/kubelet/pods/c7e50fe5-0dd0-4f59-bc5f-99199500790b/volumes/kubernetes.io~projected/kube-api-access-7n2qg major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7e50fe5-0dd0-4f59-bc5f-99199500790b/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/c7e50fe5-0dd0-4f59-bc5f-99199500790b/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7e50fe5-0dd0-4f59-bc5f-99199500790b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/c7e50fe5-0dd0-4f59-bc5f-99199500790b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7e50fe5-0dd0-4f59-bc5f-99199500790b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/c7e50fe5-0dd0-4f59-bc5f-99199500790b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7e50fe5-0dd0-4f59-bc5f-99199500790b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/c7e50fe5-0dd0-4f59-bc5f-99199500790b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7e50fe5-0dd0-4f59-bc5f-99199500790b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/c7e50fe5-0dd0-4f59-bc5f-99199500790b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7e50fe5-0dd0-4f59-bc5f-99199500790b/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/c7e50fe5-0dd0-4f59-bc5f-99199500790b/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca39ebc9-da71-43c2-893f-c033684c9399/volumes/kubernetes.io~projected/kube-api-access-gpc4k:{mountpoint:/var/lib/kubelet/pods/ca39ebc9-da71-43c2-893f-c033684c9399/volumes/kubernetes.io~projected/kube-api-access-gpc4k major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca39ebc9-da71-43c2-893f-c033684c9399/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/ca39ebc9-da71-43c2-893f-c033684c9399/volumes/kubernetes.io~secret/metrics-certs major:0 minor:403 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6bc2cf5-3e45-4217-af83-38df92cfee7f/volumes/kubernetes.io~projected/kube-api-access-rz2hs:{mountpoint:/var/lib/kubelet/pods/f6bc2cf5-3e45-4217-af83-38df92cfee7f/volumes/kubernetes.io~projected/kube-api-access-rz2hs major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6bc2cf5-3e45-4217-af83-38df92cfee7f/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/f6bc2cf5-3e45-4217-af83-38df92cfee7f/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f926eeff-6488-48c5-9e97-269a6d79dee1/volumes/kubernetes.io~projected/kube-api-access-x4545:{mountpoint:/var/lib/kubelet/pods/f926eeff-6488-48c5-9e97-269a6d79dee1/volumes/kubernetes.io~projected/kube-api-access-x4545 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f926eeff-6488-48c5-9e97-269a6d79dee1/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/f926eeff-6488-48c5-9e97-269a6d79dee1/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f951b540-0179-474c-8a2c-7ee804f9d630/volumes/kubernetes.io~projected/kube-api-access-ng4gr:{mountpoint:/var/lib/kubelet/pods/f951b540-0179-474c-8a2c-7ee804f9d630/volumes/kubernetes.io~projected/kube-api-access-ng4gr major:0 minor:409 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/facedcd2-6b70-4a3b-980c-04dceba2ef0d/volumes/kubernetes.io~projected/kube-api-access-ttk6z:{mountpoint:/var/lib/kubelet/pods/facedcd2-6b70-4a3b-980c-04dceba2ef0d/volumes/kubernetes.io~projected/kube-api-access-ttk6z major:0 minor:58 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/5c5e50e918496ef5ed81ce65c37cac7ae43182c193605a51782d4ab8f4511b96/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/67f5fef961aeddd3e0163c3208282b28d8ba26206df44c32bcb5e7696a2f0f9c/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/d690d3c2d60de8de45af1b4288af089318639c0e018c75e769c1873495a17f61/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/962163daffa6f1f2e44265f0febb3bca0960957399fd734175ea12792ee248d2/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/1f26acd3c8c6a696828935795d5ca5f4747d8c3a4d6c960c15082d8688810909/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/9b26682bb10a77cbfe6cff175ce05e87ecc96820e43dc9a05b7e6b49e1955d6f/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/6e3694e23adc88c833fda5aba3f7462f669f3bb6c10e5095ae7a4d3b246d52db/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/124d9cad454ebfbf6aabce1ca36194e2f9af295bd1ecff1abafc833e05548fde/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/8263b8e0ad04cbea7772c5527dbf37d80f3f0f8453bb48bff8a863f07c7e5878/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/716964769ace616fe0a60a6ef34f17f5be0854e55d946e2d3e9b3513409d063a/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/e4f7f40697d25246018508bb1a5362e378c5a1588ac8b19f1619ac7c9ccba976/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/40c8c5543bb26920de441ef59f1f4bebc9a0badc43f24581484e8bb7b59da402/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/ecf9de2f53931bf983bcd40002e760b974ea4d44c3f3e4d9189d1c8bb8cf875f/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/9d0bb54141602099624c3e5da4e8702e85ec71ef6316dc57e543c4395ae0d1d6/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/aea78be070c99c238f8eb53e196425d1743ba62f06d8be12a30d8d305e5d4e7d/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/ce672f7360cdfa04aa515a8f6f5ef1cabdd79fc213fefcad24cba92fa675e813/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/a430ecad0a2168771481bd49af65c128982ac8a057fd5ba6b88095bbdcb74c86/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/0db7064a207624fcbb31d516de7a5ae7c921a64f3620c4915b5e7a435a0b02bb/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/bfaa8b51d63c108575943add9d5760a0ea3aa5e88618ef0d4b6137d8f669eab2/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/be9f0b35637b9359d445e13c8479e44c91860cba7fddabc3f7048bcb1315eb34/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/b4cd6c2d175afb5950af831722f155ec3e758d9e60944b3e47ba72cb3793989f/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/672ea2b94881a7990ed879585dbfc5378f96e8b1062115f92dbdf08fa1f2a5af/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/b8240fc24115cac8d1065a9a377b3ed20195f31639e6b61f5a08a0e8b420082f/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/f6c64635ccc72496eac67a6ea0d54ffbf396a965b3b23d963e5adc1908b2373b/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/ebd0853e0657d3e6a873997d8aa145a5c5ef8d24ae03fb88a6809b392334ff7a/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/21c3f4c06fccb6eb0dfa47379df60173dd36da0af8f7b619da24c6389a4a49b7/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/ad44e52aa8af8c6955d00fe3a7c376a63f0ea7d0d317e769e1e95c11b26125f1/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/964eb005b5f4dfa369a749cd769f559e32e992399b47431bea603933ae82f7b3/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/fda44edf15ed249fee1360713bcae5f0b4233647a1b72a201587471af67b74f9/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/aa22b40a50a6048ea7bac8a81b10ac657662bfa9e2b28b1283ef1af114cabf36/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/4d70980fda4f3c277a701cad387878ce235377f4357af482a65eef7433cfc1bd/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/830cdae010d839a3766ea3f3e14ef18b9b49e051ec6f456d7b15c7ad971d0a80/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/0aba46a7047ed280a99ebce437f67bf0b7527a2b0b1b20b78a0e36c7a87ed8f1/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/0dd22bcdf41eb05d320570fdff9268ac8f49384a4c88abfc9554fff728b0f3bd/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/64d57dced6d33439b9d0256c958d7dfae37edd760d5f82a1efe07375535c4f95/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/477878fa584d9ced2290aac9c0b7e75ba9084d411774ed51f51b7161d51020aa/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/d4bcbe909796dd8751768911880eff9901157e8d05e559c4da250e4450c35bb8/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/8d827c5b24301c18af2a1008dda5cdea026b629c958cbc2851633fc0807f98bf/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/682be66c1458ec063f9f1b308b487b6df888d8ee2b323f18c69d132c1baf6c01/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/02af5e2041a3c691df1d3a8aa8d4c2fefd671b78495a2f73ace75053803ab2c1/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/bcec77b55dc46cb2600b666967e6dd2663cc5d8d1d1cfab1bd2a274751e3b6f0/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/64c19d36c2184ec018753c724db7e24739727b3b9309ad9531c685d3f5acbcd7/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/6fa074119bef127ab4ef24c7b0225a2c3340242a7d61bbfeaa8f56cb0f941627/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/9ea6222e82b936b84c0d4e87613c871d70d725d64ee697844a007884cead9f18/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/3f83dfa948f93295263560c1dffc8d900049b80e954b71704d88c7b9e3298a1f/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/bca6fc8b18e87db41fd2116bdadd80b7d1e9e31977b88ac8dd1fc3a5f538e8e0/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/161ab964cd48dec043c3857d416514a223effc3232d224771880fadfcec2ca91/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/65ced24cfa4b41cc9f93348d100f1824567801b7459212d16aa0cebadb1573da/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/396f4c40ead0fd87387b47a92985af16d71188b967e959f5c1a94daae925695c/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/878e6db1288c54752fa5541781caad440bec8b039f14ffabcd34fb81509dc89f/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/0e945452b15821a7c9af32e92c1f9d321d5cc54c1186483734de2825624c1f7a/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/144e5be75408a9bdb4e3a976b801d9f9eb7d152417763fb7f2543956dd9f957c/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/326b6dfea358e0d2cdab93b795fdbb329f8159e710e105a4600dd97fd380ce6a/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/40b29de5848cf954f9e4c95b314e5b47b04a48286436b0a2a3daca39afd8634f/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/c8d59fb85d93013587cca9c6b669ca05eb198d8f9f2483427be2f0a13b110c09/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/b067ad5855980d6cdb0f11758ce04a70f2351bdcded26d24cb0a2e3bf6cd0c17/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/447bad4657a810d0c8bd44d5d1b8f7a76961b110add13b4d2c3c56b281cf9ad3/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-418:{mountpoint:/var/lib/containers/storage/overlay/6be72c4b8d9d207b4d9128df81781866dadc1899cc9bbeec7fa1c63217694ccd/merged major:0 minor:418 fsType:overlay blockSize:0} overlay_0-420:{mountpoint:/var/lib/containers/storage/overlay/29eb44963d24ba01f6be603e23080cecf9381e3cf420eaf1dd72ec8eb32e1c9f/merged major:0 minor:420 fsType:overlay blockSize:0} overlay_0-422:{mountpoint:/var/lib/containers/storage/overlay/576b7349a1bff279c1ec02780390d9a5a55620b37ae4cea7476d86026e5e3689/merged major:0 minor:422 fsType:overlay blockSize:0} overlay_0-424:{mountpoint:/var/lib/containers/storage/overlay/111e73ba10be82f454e1227f107ae4958898833ab27528a000bcbdaf5fa325f2/merged major:0 minor:424 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/dcf1468f36b6add2697fdddbb70670a2a99d1e746e69d2151694ed31f9c57b5f/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-435:{mountpoint:/var/lib/containers/storage/overlay/d8524b626ed4dc93b26874c1cfc8819085639e062b69a199789018ff992ac113/merged major:0 minor:435 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/102a2454fb521e75fa1688d99611d3ea2252a09cac66af8df3671d727b10b6c0/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/60a4125ce8c54da5ff20e59edec00cbd3da3184bb157db6f004adefb79da0510/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/4b54d45a12ad2ac2ce3f24fc6e852eb5c3dd4a4554f2b4911715e22b38f08fd6/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/56d183a0801e2a6710b4e713538f3068a0141b1536d247a31d9bca8792d737af/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/1b3768f5c53d9cca197e73b65853e5c40cd8a2ef164f28ac1217ab030c333ed5/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/7501b318330fc1d924440b48f30bb979658cde3fb46cdcc1a623bb000aee7313/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/35adb5b9daabd5040ad5687e7ff1b6178ddb8f93a97819c1359d08a9ecbd8491/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/7f20e01a9406834c9c22f2fc16ad09bbb51fe12c720ae657d02af9773ce455c3/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/ec0a1af22640a773a95505c9f937d44f0c94b710eabf993bb0261b5edf275a2a/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/8a93ab2aa5b9f83bea940e294bd5f2bf357e03929c43bf22bc7cc7e6b6cf7fc7/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 19:37:31.290504 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.290110    5591 manager.go:217] Machine: {Timestamp:2026-11-03 19:37:31.289642092 +0000 UTC m=+0.061935102 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099650 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b8201bdaa17c40acbee899fdb840b SystemUUID:ec2b8201-bdaa-17c4-0acb-ee899fdb840b BootID:f4b3fa21-120f-4523-9e14-9e963575c8f0 Filesystems:[{Device:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:282 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-435 DeviceMajor:0 DeviceMinor:435 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:285 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8771d72e-12a3-429f-b458-e22298238393/volumes/kubernetes.io~projected/kube-api-access-5sgbt DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/300a6502-6932-49fe-936a-ba0c16978c94/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:264 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/803910f1-a1f3-4316-ada9-1890e46bbda1/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:272 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c3e81c5328eec5ff651a59728145ec14d13cc2e0bd31f41d97f17321fae9a3e3/userdata/shm DeviceMajor:0 DeviceMinor:276 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f11f7ea9334844e8aadb4a2b85717570432be27063013dd93e6846a27bc4bf14/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65b590ed-e1a5-467f-9283-7c875dd9ee24/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57562e83005ea47370f2c057b8116aa6f5e7fd92322963cdc16c9849d4ca87cd/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1cd0db81-8027-4c6a-b33c-3b4545351eac/volumes/kubernetes.io~projected/kube-api-access-gvddx DeviceMajor:0 DeviceMinor:230 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65b590ed-e1a5-467f-9283-7c875dd9ee24/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:289 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:290 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:287 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/77de5d1cd3a6ba081b6080676f08c8275491705c308f21e922697ae3ed8f06dc/userdata/shm DeviceMajor:0 DeviceMinor:305 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:289 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f45ff49a9cf6737e1a1324c65dd202171c2106d3b3d10b63706762b06b9b7cf4/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-420 DeviceMajor:0 DeviceMinor:420 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65b590ed-e1a5-467f-9283-7c875dd9ee24/volumes/kubernetes.io~projected/kube-api-access-vgdxj DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ac8ac4b-fef2-475d-a40b-9db77cded4d3/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:283 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-422 DeviceMajor:0 DeviceMinor:422 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/300a6502-6932-49fe-936a-ba0c16978c94/volumes/kubernetes.io~projected/kube-api-access-x8hqp DeviceMajor:0 DeviceMinor:270 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:286 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a7f41c5228d6776608f44fd25798b14838c46bc7bdb1a5f798a7071d15e8c35/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1cd0db81-8027-4c6a-b33c-3b4545351eac/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:229 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7e50fe5-0dd0-4f59-bc5f-99199500790b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:311 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/88a27c5e-de31-4090-be4d-6b8d6551e4c7/volumes/kubernetes.io~projected/kube-api-access-5nx7w DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ca39ebc9-da71-43c2-893f-c033684c9399/volumes/kubernetes.io~projected/kube-api-access-gpc4k DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f926eeff-6488-48c5-9e97-269a6d79dee1/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca5a374ce20ae82ec77cdcd80570f06fe84262dbb9729cd8a71a5c77f1bf9373/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/32452ae6-08d2-4fb2-863d-72dc09d4c0a2/volumes/kubernetes.io~projected/kube-api-access-dgxw5 DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c27ff198d55b17c368881572802b452da4123f9e871a6827814ff5a986e86b8/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7e50fe5-0dd0-4f59-bc5f-99199500790b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:310 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c7e50fe5-0dd0-4f59-bc5f-99199500790b/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:312 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c7e50fe5-0dd0-4f59-bc5f-99199500790b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:313 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/055215b4-9de9-494c-bbff-288cc97baf2f/volumes/kubernetes.io~projected/kube-api-access-66pfk DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/630922bd-4efc-4b44-96d1-fa55bce60e21/volumes/kubernetes.io~projected/kube-api-access-jktld DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62f8eb6fd8dcd0e75732103842fb7277ccb2358ace74075dc8fa03dc3a13af04/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/300a6502-6932-49fe-936a-ba0c16978c94/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:269 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f6bc2cf5-3e45-4217-af83-38df92cfee7f/volumes/kubernetes.io~projected/kube-api-access-rz2hs DeviceMajor:0 DeviceMinor:350 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5672d069f537ac87a1d7070bb4a617199f7a10c736d6206a803d2a3273952c9a/userdata/shm DeviceMajor:0 DeviceMinor:274 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f951b540-0179-474c-8a2c-7ee804f9d630/volumes/kubernetes.io~projected/kube-api-access-ng4gr DeviceMajor:0 DeviceMinor:409 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7e50fe5-0dd0-4f59-bc5f-99199500790b/volumes/kubernetes.io~projected/kube-api-access-7n2qg DeviceMajor:0 DeviceMinor:318 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:289 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e6bc54351c35f29c80caf98aa51d0e66f770cc73a6380359128f72ab58b65d8/userdata/shm DeviceMajor:0 DeviceMinor:410 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7e50fe5-0dd0-4f59-bc5f-99199500790b/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:317 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71c679be-e63f-4ad4-a1b2-5474ec4fa125/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f926eeff-6488-48c5-9e97-269a6d79dee1/volumes/kubernetes.io~projected/kube-api-access-x4545 DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e4d01dcd5adba2d4060227e841b164fc8265e16eec03d9d866e829e9e8e8af84/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8edd2f162297442f9f77e5ba98660f4c491934551e31553c054c9eecdd98d5b1/userdata/shm DeviceMajor:0 DeviceMinor:97 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/71c679be-e63f-4ad4-a1b2-5474ec4fa125/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/803910f1-a1f3-4316-ada9-1890e46bbda1/volumes/kubernetes.io~projected/kube-api-access-q6rwg DeviceMajor:0 DeviceMinor:273 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:289 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~projected/kube-api-access-tmggx DeviceMajor:0 DeviceMinor:291 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-418 DeviceMajor:0 DeviceMinor:418 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/71c679be-e63f-4ad4-a1b2-5474ec4fa125/volumes/kubernetes.io~projected/kube-api-access-4zmfm DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dc002ae6adfc9db21f86304ef7cef337adb0de17d39be256b6634f686cf10d88/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:290 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da480f56accb6891c3c0d00769c98771f2bd08c62e5ec777516d31555cb5a27d/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0eaf6115f4579620b0f14c3598a9d52a4cd6355129bde16bac809b7d568be2ec/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b4a42db6-0377-4a5a-94ab-be3f82b332aa/volumes/kubernetes.io~projected/kube-api-access-tzmv5 DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1cd0db81-8027-4c6a-b33c-3b4545351eac/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/803910f1-a1f3-4316-ada9-1890e46bbda1/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:271 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:284 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d8fa1f9d28b5fd9c66cab82f903427748428dc067259baa487701b7c3e950780/userdata/shm DeviceMajor:0 DeviceMinor:412 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-424 DeviceMajor:0 DeviceMinor:424 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/facedcd2-6b70-4a3b-980c-04dceba2ef0d/volumes/kubernetes.io~projected/kube-api-access-ttk6z DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0f241ca48f494c140d074ff6fcfb46095a2ca4874c8c33c1d31069343b99f3dd/userdata/shm DeviceMajor:0 DeviceMinor:351 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3b53f7107aff24d23d341b7d5189301d877d896ed6e3fd7f52eed5a7b978636/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/297782fc2eb6ce8a7dd368a98ac3593c216345a3f5ff82942c8d18edba787c48/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8771d72e-12a3-429f-b458-e22298238393/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f6bc2cf5-3e45-4217-af83-38df92cfee7f/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:346 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ac8ac4b-fef2-475d-a40b-9db77cded4d3/volumes/kubernetes.io~projected/kube-api-access-jp6g4 DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91ac845a4e0fd4d48706806d9296092656f225636a066a05db0d42d15dee5ec3/userdata/shm DeviceMajor:0 DeviceMinor:414 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48770e51f689ef669cbfb99db1b67bb09f19a6ca313a5d7c90707b36979f2dc2/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b1cda986-f386-462d-9956-acc2a4de6e32/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:288 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c7e50fe5-0dd0-4f59-bc5f-99199500790b/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:314 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ca39ebc9-da71-43c2-893f-c033684c9399/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:403 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/98d32fbe-5011-47a8-b937-725a8cad95a4/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:408 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0eaf6115f457962 MacAddress:9a:77:52:af:8a:1e Speed:10000 Mtu:8901} {Name:0f241ca48f494c1 MacAddress:2a:7b:71:67:56:5a Speed:10000 Mtu:8901} {Name:3e6bc54351c35f2 MacAddress:7e:f2:4b:e7:b2:e6 Speed:10000 Mtu:8901} {Name:48770e51f689ef6 MacAddress:6a:ff:d1:50:f7:5c Speed:10000 Mtu:8901} {Name:5672d069f537ac8 MacAddress:8a:ed:af:82:45:d1 Speed:10000 Mtu:8901} {Name:57562e83005ea47 MacAddress:46:8c:1b:01:22:65 Speed:10000 Mtu:8901} {Name:62f8eb6fd8dcd0e MacAddress:2e:44:99:7b:88:6d Speed:10000 Mtu:8901} {Name:77de5d1cd3a6ba0 MacAddress:fe:ae:92:1b:6a:ca Speed:10000 Mtu:8901} {Name:8c27ff198d55b17 MacAddress:de:ab:e7:ef:7b:86 Speed:10000 Mtu:8901} {Name:91ac845a4e0fd4d MacAddress:5a:1d:98:e3:b9:a3 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:db:50:90:7f Speed:0 Mtu:9001} {Name:br-int MacAddress:e6:8b:e1:71:a8:1e Speed:0 Mtu:8901} {Name:d8fa1f9d28b5fd9 MacAddress:9e:80:ce:fa:f3:92 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:db:50:90:7f Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:6e:8f:7c:e1:8b:1a Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:f6:b6:a7:44:5a:86 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 19:37:31.290696 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.290508    5591 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 19:37:31.290696 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.290553    5591 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 19:37:31.290926 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.290906    5591 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 19:37:31.291079 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.290926    5591 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-206.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 19:37:31.291122 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.291081    5591 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 19:37:31.291122 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.291090    5591 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 19:37:31.291122 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.291104    5591 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:37:31.291198 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.291123    5591 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 19:37:31.291198 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.291219    5591 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 19:37:31.291424 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.291414    5591 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 19:37:31.291540 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.291532    5591 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 19:37:31.291587 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.291580    5591 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 19:37:31.291615 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.291595    5591 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 19:37:31.291615 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.291604    5591 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 19:37:31.291615 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.291614    5591 kubelet.go:405] "Adding apiserver pod source"
Nov 03 19:37:31.291703 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.291623    5591 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 19:37:31.294353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.293318    5591 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 19:37:31.294353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.293339    5591 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:37:31.294705 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.294596    5591 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 19:37:31.297572 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.297555    5591 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 19:37:31.297572 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.297569    5591 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 19:37:31.297711 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.297674    5591 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 19:37:31.297873 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.297862    5591 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 19:37:31.297904 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.297891    5591 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 19:37:31.297992 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.297983    5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 19:37:31.298024 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.297998    5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 19:37:31.298024 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.298005    5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 19:37:31.298024 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.298011    5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 19:37:31.298024 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.298016    5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 19:37:31.298024 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.298021    5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 19:37:31.298024 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.298027    5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 19:37:31.298331 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.298033    5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 19:37:31.298331 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.298040    5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 19:37:31.298331 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.298045    5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 19:37:31.298331 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.298053    5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 19:37:31.298331 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.298062    5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 19:37:31.298331 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.298086    5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 19:37:31.298331 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.298093    5591 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 19:37:31.301675 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.301662    5591 server.go:1263] "Started kubelet"
Nov 03 19:37:31.301852 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.301778    5591 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 19:37:31.301904 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.301880    5591 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 19:37:31.301956 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.301794    5591 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 19:37:31.302261 ip-10-0-143-206 systemd[1]: Started Kubernetes Kubelet.
Nov 03 19:37:31.303612 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.303245    5591 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 19:37:31.303612 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.303603    5591 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 19:37:31.307321 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.307301    5591 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 19:37:31.310671 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.310625    5591 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 19:37:31.310671 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.310669    5591 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 19:37:31.310815 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:37:31.310759    5591 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-206.us-east-2.compute.internal\" not found"
Nov 03 19:37:31.310866 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.310821    5591 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 19:37:31.311236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.311198    5591 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 19:37:31.311540 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.311500    5591 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:31:21 +0000 UTC" deadline="2028-06-01 06:45:36.906331544 +0000 UTC"
Nov 03 19:37:31.311626 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.311538    5591 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13811h8m5.594796274s"
Nov 03 19:37:31.312925 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.312898    5591 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:37:31.316045 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.315992    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" volumeName="kubernetes.io/empty-dir/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-config-out" seLinuxMountContext=""
Nov 03 19:37:31.316147 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316047    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1cda986-f386-462d-9956-acc2a4de6e32" volumeName="kubernetes.io/configmap/b1cda986-f386-462d-9956-acc2a4de6e32-konnectivity-ca" seLinuxMountContext=""
Nov 03 19:37:31.316147 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316063    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca39ebc9-da71-43c2-893f-c033684c9399" volumeName="kubernetes.io/projected/ca39ebc9-da71-43c2-893f-c033684c9399-kube-api-access-gpc4k" seLinuxMountContext=""
Nov 03 19:37:31.316147 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316075    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="300a6502-6932-49fe-936a-ba0c16978c94" volumeName="kubernetes.io/configmap/300a6502-6932-49fe-936a-ba0c16978c94-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Nov 03 19:37:31.316147 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316086    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1cda986-f386-462d-9956-acc2a4de6e32" volumeName="kubernetes.io/secret/b1cda986-f386-462d-9956-acc2a4de6e32-agent-certs" seLinuxMountContext=""
Nov 03 19:37:31.316147 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316097    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6bc2cf5-3e45-4217-af83-38df92cfee7f" volumeName="kubernetes.io/projected/f6bc2cf5-3e45-4217-af83-38df92cfee7f-kube-api-access-rz2hs" seLinuxMountContext=""
Nov 03 19:37:31.316147 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316107    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f926eeff-6488-48c5-9e97-269a6d79dee1" volumeName="kubernetes.io/secret/f926eeff-6488-48c5-9e97-269a6d79dee1-cert" seLinuxMountContext=""
Nov 03 19:37:31.316147 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316120    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32452ae6-08d2-4fb2-863d-72dc09d4c0a2" volumeName="kubernetes.io/configmap/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-multus-daemon-config" seLinuxMountContext=""
Nov 03 19:37:31.316147 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316137    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" volumeName="kubernetes.io/empty-dir/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-alertmanager-main-db" seLinuxMountContext=""
Nov 03 19:37:31.316147 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316149    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" volumeName="kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316160    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" volumeName="kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-web-config" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316190    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" volumeName="kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-cluster-tls-config" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316225    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7e50fe5-0dd0-4f59-bc5f-99199500790b" volumeName="kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316239    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1cd0db81-8027-4c6a-b33c-3b4545351eac" volumeName="kubernetes.io/projected/1cd0db81-8027-4c6a-b33c-3b4545351eac-kube-api-access-gvddx" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316270    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="facedcd2-6b70-4a3b-980c-04dceba2ef0d" volumeName="kubernetes.io/projected/facedcd2-6b70-4a3b-980c-04dceba2ef0d-kube-api-access-ttk6z" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316282    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1cd0db81-8027-4c6a-b33c-3b4545351eac" volumeName="kubernetes.io/secret/1cd0db81-8027-4c6a-b33c-3b4545351eac-prometheus-operator-tls" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316295    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="300a6502-6932-49fe-936a-ba0c16978c94" volumeName="kubernetes.io/projected/300a6502-6932-49fe-936a-ba0c16978c94-kube-api-access-x8hqp" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316307    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="055215b4-9de9-494c-bbff-288cc97baf2f" volumeName="kubernetes.io/projected/055215b4-9de9-494c-bbff-288cc97baf2f-kube-api-access-66pfk" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316332    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ac8ac4b-fef2-475d-a40b-9db77cded4d3" volumeName="kubernetes.io/secret/3ac8ac4b-fef2-475d-a40b-9db77cded4d3-metrics-tls" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316348    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65b590ed-e1a5-467f-9283-7c875dd9ee24" volumeName="kubernetes.io/configmap/65b590ed-e1a5-467f-9283-7c875dd9ee24-env-overrides" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316360    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8771d72e-12a3-429f-b458-e22298238393" volumeName="kubernetes.io/empty-dir/8771d72e-12a3-429f-b458-e22298238393-data-volume" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316372    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4a42db6-0377-4a5a-94ab-be3f82b332aa" volumeName="kubernetes.io/projected/b4a42db6-0377-4a5a-94ab-be3f82b332aa-kube-api-access-tzmv5" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316386    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="300a6502-6932-49fe-936a-ba0c16978c94" volumeName="kubernetes.io/empty-dir/300a6502-6932-49fe-936a-ba0c16978c94-volume-directive-shadow" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316398    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" volumeName="kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-main-tls" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316409    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4a42db6-0377-4a5a-94ab-be3f82b332aa" volumeName="kubernetes.io/configmap/b4a42db6-0377-4a5a-94ab-be3f82b332aa-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316422    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="300a6502-6932-49fe-936a-ba0c16978c94" volumeName="kubernetes.io/configmap/300a6502-6932-49fe-936a-ba0c16978c94-metrics-client-ca" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316434    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ac8ac4b-fef2-475d-a40b-9db77cded4d3" volumeName="kubernetes.io/configmap/3ac8ac4b-fef2-475d-a40b-9db77cded4d3-config-volume" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316445    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ac8ac4b-fef2-475d-a40b-9db77cded4d3" volumeName="kubernetes.io/empty-dir/3ac8ac4b-fef2-475d-a40b-9db77cded4d3-tmp-dir" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316463    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="630922bd-4efc-4b44-96d1-fa55bce60e21" volumeName="kubernetes.io/projected/630922bd-4efc-4b44-96d1-fa55bce60e21-kube-api-access-jktld" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316474    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65b590ed-e1a5-467f-9283-7c875dd9ee24" volumeName="kubernetes.io/configmap/65b590ed-e1a5-467f-9283-7c875dd9ee24-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316486    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65b590ed-e1a5-467f-9283-7c875dd9ee24" volumeName="kubernetes.io/projected/65b590ed-e1a5-467f-9283-7c875dd9ee24-kube-api-access-vgdxj" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316507    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="055215b4-9de9-494c-bbff-288cc97baf2f" volumeName="kubernetes.io/configmap/055215b4-9de9-494c-bbff-288cc97baf2f-iptables-alerter-script" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316519    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="300a6502-6932-49fe-936a-ba0c16978c94" volumeName="kubernetes.io/secret/300a6502-6932-49fe-936a-ba0c16978c94-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316532    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="803910f1-a1f3-4316-ada9-1890e46bbda1" volumeName="kubernetes.io/configmap/803910f1-a1f3-4316-ada9-1890e46bbda1-metrics-client-ca" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316543    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" volumeName="kubernetes.io/configmap/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-metrics-client-ca" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316556    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" volumeName="kubernetes.io/configmap/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316567    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" volumeName="kubernetes.io/projected/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-tls-assets" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316581    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7e50fe5-0dd0-4f59-bc5f-99199500790b" volumeName="kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-tls" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316594    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1cd0db81-8027-4c6a-b33c-3b4545351eac" volumeName="kubernetes.io/configmap/1cd0db81-8027-4c6a-b33c-3b4545351eac-metrics-client-ca" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316607    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71c679be-e63f-4ad4-a1b2-5474ec4fa125" volumeName="kubernetes.io/empty-dir/71c679be-e63f-4ad4-a1b2-5474ec4fa125-tmp" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316619    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" volumeName="kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316630    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98d32fbe-5011-47a8-b937-725a8cad95a4" volumeName="kubernetes.io/secret/98d32fbe-5011-47a8-b937-725a8cad95a4-original-pull-secret" seLinuxMountContext=""
Nov 03 19:37:31.316624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316642    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7e50fe5-0dd0-4f59-bc5f-99199500790b" volumeName="kubernetes.io/configmap/c7e50fe5-0dd0-4f59-bc5f-99199500790b-metrics-client-ca" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316653    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65b590ed-e1a5-467f-9283-7c875dd9ee24" volumeName="kubernetes.io/configmap/65b590ed-e1a5-467f-9283-7c875dd9ee24-ovnkube-config" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316667    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="803910f1-a1f3-4316-ada9-1890e46bbda1" volumeName="kubernetes.io/configmap/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316678    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88a27c5e-de31-4090-be4d-6b8d6551e4c7" volumeName="kubernetes.io/configmap/88a27c5e-de31-4090-be4d-6b8d6551e4c7-serviceca" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316690    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7e50fe5-0dd0-4f59-bc5f-99199500790b" volumeName="kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316702    5591 factory.go:55] Registering systemd factory
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316717    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8771d72e-12a3-429f-b458-e22298238393" volumeName="kubernetes.io/projected/8771d72e-12a3-429f-b458-e22298238393-kube-api-access-5sgbt" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316731    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" volumeName="kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316743    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4a42db6-0377-4a5a-94ab-be3f82b332aa" volumeName="kubernetes.io/configmap/b4a42db6-0377-4a5a-94ab-be3f82b332aa-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316756    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7e50fe5-0dd0-4f59-bc5f-99199500790b" volumeName="kubernetes.io/empty-dir/c7e50fe5-0dd0-4f59-bc5f-99199500790b-tmp" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316779    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1cd0db81-8027-4c6a-b33c-3b4545351eac" volumeName="kubernetes.io/secret/1cd0db81-8027-4c6a-b33c-3b4545351eac-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316790    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32452ae6-08d2-4fb2-863d-72dc09d4c0a2" volumeName="kubernetes.io/projected/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-kube-api-access-dgxw5" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316802    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65b590ed-e1a5-467f-9283-7c875dd9ee24" volumeName="kubernetes.io/secret/65b590ed-e1a5-467f-9283-7c875dd9ee24-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316815    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" volumeName="kubernetes.io/projected/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-kube-api-access-tmggx" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316827    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" volumeName="kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-config-volume" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316870    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7e50fe5-0dd0-4f59-bc5f-99199500790b" volumeName="kubernetes.io/projected/c7e50fe5-0dd0-4f59-bc5f-99199500790b-kube-api-access-7n2qg" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316883    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7e50fe5-0dd0-4f59-bc5f-99199500790b" volumeName="kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-grpc-tls" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316896    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca39ebc9-da71-43c2-893f-c033684c9399" volumeName="kubernetes.io/secret/ca39ebc9-da71-43c2-893f-c033684c9399-metrics-certs" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316908    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32452ae6-08d2-4fb2-863d-72dc09d4c0a2" volumeName="kubernetes.io/configmap/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-cni-binary-copy" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316919    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71c679be-e63f-4ad4-a1b2-5474ec4fa125" volumeName="kubernetes.io/projected/71c679be-e63f-4ad4-a1b2-5474ec4fa125-kube-api-access-4zmfm" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316932    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="803910f1-a1f3-4316-ada9-1890e46bbda1" volumeName="kubernetes.io/empty-dir/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-textfile" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316945    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="803910f1-a1f3-4316-ada9-1890e46bbda1" volumeName="kubernetes.io/secret/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316956    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8771d72e-12a3-429f-b458-e22298238393" volumeName="kubernetes.io/secret/8771d72e-12a3-429f-b458-e22298238393-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316968    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6bc2cf5-3e45-4217-af83-38df92cfee7f" volumeName="kubernetes.io/secret/f6bc2cf5-3e45-4217-af83-38df92cfee7f-monitoring-plugin-cert" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316979    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f926eeff-6488-48c5-9e97-269a6d79dee1" volumeName="kubernetes.io/projected/f926eeff-6488-48c5-9e97-269a6d79dee1-kube-api-access-x4545" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316991    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="300a6502-6932-49fe-936a-ba0c16978c94" volumeName="kubernetes.io/secret/300a6502-6932-49fe-936a-ba0c16978c94-kube-state-metrics-tls" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.317006    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ac8ac4b-fef2-475d-a40b-9db77cded4d3" volumeName="kubernetes.io/projected/3ac8ac4b-fef2-475d-a40b-9db77cded4d3-kube-api-access-jp6g4" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.317018    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="630922bd-4efc-4b44-96d1-fa55bce60e21" volumeName="kubernetes.io/empty-dir/630922bd-4efc-4b44-96d1-fa55bce60e21-tmp-dir" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.317031    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="803910f1-a1f3-4316-ada9-1890e46bbda1" volumeName="kubernetes.io/secret/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-tls" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.317043    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7e50fe5-0dd0-4f59-bc5f-99199500790b" volumeName="kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.317054    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88a27c5e-de31-4090-be4d-6b8d6551e4c7" volumeName="kubernetes.io/projected/88a27c5e-de31-4090-be4d-6b8d6551e4c7-kube-api-access-5nx7w" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.317067    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4a42db6-0377-4a5a-94ab-be3f82b332aa" volumeName="kubernetes.io/configmap/b4a42db6-0377-4a5a-94ab-be3f82b332aa-cni-binary-copy" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.317079    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7e50fe5-0dd0-4f59-bc5f-99199500790b" volumeName="kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.317091    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f951b540-0179-474c-8a2c-7ee804f9d630" volumeName="kubernetes.io/projected/f951b540-0179-474c-8a2c-7ee804f9d630-kube-api-access-ng4gr" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.317103    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71c679be-e63f-4ad4-a1b2-5474ec4fa125" volumeName="kubernetes.io/empty-dir/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-tuned" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.317116    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="803910f1-a1f3-4316-ada9-1890e46bbda1" volumeName="kubernetes.io/projected/803910f1-a1f3-4316-ada9-1890e46bbda1-kube-api-access-q6rwg" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.317128    5591 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8771d72e-12a3-429f-b458-e22298238393" volumeName="kubernetes.io/configmap/8771d72e-12a3-429f-b458-e22298238393-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.317142    5591 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.317151    5591 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.316722    5591 factory.go:223] Registration of the systemd container factory successfully
Nov 03 19:37:31.318118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.317325    5591 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:37:31.319631 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.318299    5591 factory.go:153] Registering CRI-O factory
Nov 03 19:37:31.319631 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.318313    5591 factory.go:223] Registration of the crio container factory successfully
Nov 03 19:37:31.319631 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.318385    5591 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 19:37:31.319631 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.318406    5591 factory.go:103] Registering Raw factory
Nov 03 19:37:31.319631 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.318420    5591 manager.go:1199] Started watching for new ooms in manager
Nov 03 19:37:31.319631 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.318549    5591 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:37:31.319631 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.318859    5591 manager.go:319] Starting recovery of all containers
Nov 03 19:37:31.324412 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.324391    5591 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 19:37:31.329032 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.328961    5591 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 19:37:31.340924 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.340897    5591 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 19:37:31.341036 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.340933    5591 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 19:37:31.341036 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.340955    5591 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 19:37:31.341036 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:37:31.340990    5591 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 19:37:31.345155 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.345118    5591 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 19:37:31.348184 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.348149    5591 generic.go:356] "Generic (PLEG): container finished" podID="6c67d5fc48584d861d9d388770293f30" containerID="3c4f638977774a2da8ebe5666d3201fa97be208b0bd50951b9ab288d2f28ffad" exitCode=0
Nov 03 19:37:31.349567 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.349546    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 19:37:31.349648 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.349548    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 19:37:31.369185 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.369153    5591 generic.go:356] "Generic (PLEG): container finished" podID="803910f1-a1f3-4316-ada9-1890e46bbda1" containerID="17c788662e3bd5d72eef80a87bb40ca5ea53b11c218d5473cd2aeee2274dbcd3" exitCode=0
Nov 03 19:37:31.378679 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.378655    5591 generic.go:356] "Generic (PLEG): container finished" podID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerID="5d900c564cd425ff4cebb7144fdc0d60bf5953d447cc834619b884d13dce2b77" exitCode=0
Nov 03 19:37:31.390925 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.390896    5591 generic.go:356] "Generic (PLEG): container finished" podID="b4a42db6-0377-4a5a-94ab-be3f82b332aa" containerID="64ec19ed181218aa44a8a488bce53a7fc0bc4ae31c1a6e1fbfbe783677805919" exitCode=0
Nov 03 19:37:31.390925 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.390922    5591 generic.go:356] "Generic (PLEG): container finished" podID="b4a42db6-0377-4a5a-94ab-be3f82b332aa" containerID="054a11272a751929b7e38eb5850975116b7823e823ce7d82266f17029e1771b8" exitCode=0
Nov 03 19:37:31.390925 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.390929    5591 generic.go:356] "Generic (PLEG): container finished" podID="b4a42db6-0377-4a5a-94ab-be3f82b332aa" containerID="dca166d961a8c0bf8f4fce68c01ef8c985a4de0c7ca472a7e7d40883aca28a47" exitCode=0
Nov 03 19:37:31.391165 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.390935    5591 generic.go:356] "Generic (PLEG): container finished" podID="b4a42db6-0377-4a5a-94ab-be3f82b332aa" containerID="cf8c2a98f6b4f3d42b63ea1fdf8902c3366e20f3052ea52259a3e6ba02564b7c" exitCode=0
Nov 03 19:37:31.391165 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.390941    5591 generic.go:356] "Generic (PLEG): container finished" podID="b4a42db6-0377-4a5a-94ab-be3f82b332aa" containerID="3afd2dbeb9ef4b7316a2063dcf6d99ccc5b44059cdbabae60f2a058477f1e803" exitCode=0
Nov 03 19:37:31.391165 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.390946    5591 generic.go:356] "Generic (PLEG): container finished" podID="b4a42db6-0377-4a5a-94ab-be3f82b332aa" containerID="16e756e1f4bba1d8fdbe23eb419c63372566a3b8e24f0c0bb25783565b787913" exitCode=0
Nov 03 19:37:31.396155 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.396131    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 19:37:31.396570 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.396542    5591 generic.go:356] "Generic (PLEG): container finished" podID="65b590ed-e1a5-467f-9283-7c875dd9ee24" containerID="026bd52efdad675243c5ef5fc94f526681bad64e10431d752cd39c325214431a" exitCode=1
Nov 03 19:37:31.425582 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.425511    5591 manager.go:324] Recovery completed
Nov 03 19:37:31.442091 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:37:31.442056    5591 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 03 19:37:31.450327 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.450306    5591 cpu_manager.go:226] "Starting" policy="none"
Nov 03 19:37:31.450327 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.450321    5591 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 19:37:31.450492 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.450338    5591 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 19:37:31.450492 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.450460    5591 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 19:37:31.450492 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.450471    5591 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 19:37:31.450492 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.450488    5591 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 19:37:31.450492 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.450494    5591 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 19:37:31.450681 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.450499    5591 policy_none.go:50] "Start"
Nov 03 19:37:31.450681 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.450507    5591 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 19:37:31.450681 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.450515    5591 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 19:37:31.450681 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.450636    5591 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 19:37:31.450681 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.450644    5591 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 19:37:31.450681 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.450649    5591 policy_none.go:44] "Start"
Nov 03 19:37:31.455150 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.455119    5591 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 19:37:31.457752 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.457740    5591 manager.go:341] "Starting Device Plugin manager"
Nov 03 19:37:31.458292 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:37:31.457765    5591 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 19:37:31.458292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.458294    5591 server.go:87] "Starting device plugin registration server"
Nov 03 19:37:31.458554 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.458543    5591 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 19:37:31.458617 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.458555    5591 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 19:37:31.458684 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.458666    5591 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 19:37:31.458798 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.458768    5591 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 19:37:31.458849 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.458814    5591 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 19:37:31.458849 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.458821    5591 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 19:37:31.458949 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.458864    5591 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:37:31.458796346Z","UUID":"ae51377d-a3f4-4707-99e0-83cc5b76ce12","Handler":null,"Name":"","Endpoint":""}
Nov 03 19:37:31.460364 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.460336    5591 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 19:37:31.460364 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.460358    5591 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 19:37:31.460519 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:37:31.460444    5591 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 19:37:31.559244 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.559184    5591 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 19:37:31.561578 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.561561    5591 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-206.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 19:37:31.561701 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.561594    5591 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-206.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 19:37:31.561701 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.561606    5591 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-206.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 19:37:31.561701 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.561649    5591 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:37:31.581429 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.581398    5591 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:37:31.581557 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.581483    5591 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:37:31.642952 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.642876    5591 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal"]
Nov 03 19:37:31.643585 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.643538    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal" event={"ID":"3def2722c7c58bcddedb41ddbc48c3ab","Type":"ContainerStarted","Data":"9eeb4aee2f3f207aedbf2c59014cc70609199c30fc6ecf19d39fc0f4a46c0b5f"}
Nov 03 19:37:31.643691 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.643597    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal" event={"ID":"3def2722c7c58bcddedb41ddbc48c3ab","Type":"ContainerStarted","Data":"e3b53f7107aff24d23d341b7d5189301d877d896ed6e3fd7f52eed5a7b978636"}
Nov 03 19:37:31.643691 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.643615    5591 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:37:31.643776 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.643748    5591 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:37:31.656451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.656420    5591 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:37:31.656555 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:37:31.656491    5591 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:37:31.656555 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.656425    5591 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:37:31.656555 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:37:31.656552    5591 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:37:31.726872 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.726780    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6c67d5fc48584d861d9d388770293f30-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal\" (UID: \"6c67d5fc48584d861d9d388770293f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:37:31.726872 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.726841    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c67d5fc48584d861d9d388770293f30-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal\" (UID: \"6c67d5fc48584d861d9d388770293f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:37:31.726872 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.726869    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3def2722c7c58bcddedb41ddbc48c3ab-config\") pod \"kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal\" (UID: \"3def2722c7c58bcddedb41ddbc48c3ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:37:31.827479 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.827440    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c67d5fc48584d861d9d388770293f30-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal\" (UID: \"6c67d5fc48584d861d9d388770293f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:37:31.827657 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.827484    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3def2722c7c58bcddedb41ddbc48c3ab-config\") pod \"kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal\" (UID: \"3def2722c7c58bcddedb41ddbc48c3ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:37:31.827657 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.827513    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6c67d5fc48584d861d9d388770293f30-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal\" (UID: \"6c67d5fc48584d861d9d388770293f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:37:31.827657 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.827564    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c67d5fc48584d861d9d388770293f30-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal\" (UID: \"6c67d5fc48584d861d9d388770293f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:37:31.827657 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.827566    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3def2722c7c58bcddedb41ddbc48c3ab-config\") pod \"kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal\" (UID: \"3def2722c7c58bcddedb41ddbc48c3ab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:37:31.827657 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:31.827572    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6c67d5fc48584d861d9d388770293f30-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal\" (UID: \"6c67d5fc48584d861d9d388770293f30\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:37:32.292349 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.292315    5591 apiserver.go:52] "Watching apiserver"
Nov 03 19:37:32.301426 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.301401    5591 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 19:37:32.302147 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.302125    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-vh26d","openshift-dns/node-resolver-rzn4t","openshift-multus/multus-additional-cni-plugins-q4tbd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf","openshift-monitoring/alertmanager-main-0","openshift-network-diagnostics/network-check-target-pfcnb","openshift-monitoring/node-exporter-sckcx","openshift-monitoring/prometheus-operator-9cd89b74c-rvqts","kube-system/kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-mgrnm","openshift-multus/multus-xtsbd","openshift-ovn-kubernetes/ovnkube-node-x88xn","kube-system/global-pull-secret-syncer-6q5vq","openshift-ingress-canary/ingress-canary-mksqw","openshift-image-registry/node-ca-pswxd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-206.us-east-2.compute.internal","openshift-network-operator/iptables-alerter-htzs4","openshift-insights/insights-runtime-extractor-2224c","openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk","openshift-monitoring/monitoring-plugin-77f4f76594-bnrmk","openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct","openshift-multus/network-metrics-daemon-794nj","openshift-dns/dns-default-868lx"]
Nov 03 19:37:32.307086 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.307057    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 19:37:32.307283 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.307264    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 19:37:32.307362 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.307354    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 19:37:32.307419 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.307369    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qhpjg\""
Nov 03 19:37:32.307419 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.307393    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 19:37:32.307500 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.307426    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 19:37:32.307500 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.307485    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tx8mk\""
Nov 03 19:37:32.310450 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.310430    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 19:37:32.310549 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.310465    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 19:37:32.310682 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.310667    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 19:37:32.310793 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.310779    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 19:37:32.310929 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.310909    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 19:37:32.311154 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.311135    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 19:37:32.311523 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.311504    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ndfpg\""
Nov 03 19:37:32.311523 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.311519    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 19:37:32.311678 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.311607    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lrbp4\""
Nov 03 19:37:32.311678 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.311620    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5qm8x\""
Nov 03 19:37:32.311678 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.311632    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 19:37:32.311678 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.311642    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 19:37:32.311678 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.311645    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 19:37:32.311678 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.311653    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 19:37:32.313424 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.313350    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 19:37:32.313602 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.313582    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 19:37:32.313703 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.313625    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 19:37:32.313703 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.313622    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 19:37:32.313703 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.313673    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 19:37:32.313703 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.313672    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 19:37:32.313982 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.313966    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 19:37:32.314037 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.313979    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:37:32.314037 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.314005    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 19:37:32.314128 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.314045    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 19:37:32.314128 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.314074    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rs44g\""
Nov 03 19:37:32.314128 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.314050    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vlmgx\""
Nov 03 19:37:32.314128 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.314100    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 19:37:32.314128 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.314105    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 19:37:32.314128 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.314099    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 19:37:32.314128 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.314118    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xv4gr\""
Nov 03 19:37:32.314128 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.314118    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 19:37:32.314411 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.314122    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 19:37:32.314411 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.314162    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xbkg2\""
Nov 03 19:37:32.314411 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.314272    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bkd9f\""
Nov 03 19:37:32.314502 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.314465    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jzp4r\""
Nov 03 19:37:32.314586 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.314568    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ldllm\""
Nov 03 19:37:32.315051 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.315032    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 19:37:32.315148 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.315081    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 19:37:32.315236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.315145    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 19:37:32.315236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.315155    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 19:37:32.315236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.315178    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 19:37:32.315236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.315192    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 19:37:32.315713 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.315694    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dgjzc\""
Nov 03 19:37:32.315915 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.315724    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 19:37:32.316014 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.315744    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 19:37:32.317090 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.317066    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b2f85\""
Nov 03 19:37:32.317188 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.317107    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-xc2tq\""
Nov 03 19:37:32.317188 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.317148    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 19:37:32.317606 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.317554    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 19:37:32.317701 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.317617    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 19:37:32.317701 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.317668    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 19:37:32.317800 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.317572    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lhxcg\""
Nov 03 19:37:32.317882 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.317864    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 03 19:37:32.318100 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.318084    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 19:37:32.318273 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.318243    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 03 19:37:32.318448 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.318431    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 19:37:32.318497 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.318445    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 19:37:32.318497 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.318455    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 03 19:37:32.318599 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.318510    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 19:37:32.318673 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.318655    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qg9jj\""
Nov 03 19:37:32.318723 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.318690    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 19:37:32.318770 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.318722    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 19:37:32.318819 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.318787    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 19:37:32.318864 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.318823    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 03 19:37:32.318864 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.318857    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 19:37:32.319025 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.319011    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pz944\""
Nov 03 19:37:32.319096 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.319084    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-54rzd\""
Nov 03 19:37:32.319186 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.319174    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 19:37:32.319341 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.319328    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 03 19:37:32.319412 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.319399    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 19:37:32.321365 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.321349    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 19:37:32.322720 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.322690    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 19:37:32.322855 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.322768    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 19:37:32.322968 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.322900    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 19:37:32.322968 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.322953    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 19:37:32.323229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.323193    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-nxvmw\""
Nov 03 19:37:32.323229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.323216    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 19:37:32.323345 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.323217    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-53fusnobhruq3\""
Nov 03 19:37:32.323345 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.323245    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 19:37:32.323345 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.323261    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5x9cb\""
Nov 03 19:37:32.325705 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.325692    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 19:37:32.329065 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329048    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-run-ovn-kubernetes\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.329113 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329073    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.329113 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329094    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/055215b4-9de9-494c-bbff-288cc97baf2f-iptables-alerter-script\") pod \"iptables-alerter-htzs4\" (UID: \"055215b4-9de9-494c-bbff-288cc97baf2f\") " pod="openshift-network-operator/iptables-alerter-htzs4"
Nov 03 19:37:32.329113 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329110    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/055215b4-9de9-494c-bbff-288cc97baf2f-host-slash\") pod \"iptables-alerter-htzs4\" (UID: \"055215b4-9de9-494c-bbff-288cc97baf2f\") " pod="openshift-network-operator/iptables-alerter-htzs4"
Nov 03 19:37:32.329245 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329230    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/630922bd-4efc-4b44-96d1-fa55bce60e21-tmp-dir\") pod \"node-resolver-rzn4t\" (UID: \"630922bd-4efc-4b44-96d1-fa55bce60e21\") " pod="openshift-dns/node-resolver-rzn4t"
Nov 03 19:37:32.329340 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329260    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-lib-modules\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.329340 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329279    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-cnibin\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.329340 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329299    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/055215b4-9de9-494c-bbff-288cc97baf2f-iptables-alerter-script\") pod \"iptables-alerter-htzs4\" (UID: \"055215b4-9de9-494c-bbff-288cc97baf2f\") " pod="openshift-network-operator/iptables-alerter-htzs4"
Nov 03 19:37:32.329340 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329308    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-var-lib-cni-multus\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.329489 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329341    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/88a27c5e-de31-4090-be4d-6b8d6551e4c7-serviceca\") pod \"node-ca-pswxd\" (UID: \"88a27c5e-de31-4090-be4d-6b8d6551e4c7\") " pod="openshift-image-registry/node-ca-pswxd"
Nov 03 19:37:32.329489 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329354    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/630922bd-4efc-4b44-96d1-fa55bce60e21-tmp-dir\") pod \"node-resolver-rzn4t\" (UID: \"630922bd-4efc-4b44-96d1-fa55bce60e21\") " pod="openshift-dns/node-resolver-rzn4t"
Nov 03 19:37:32.329489 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329375    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-socket-dir\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:37:32.329489 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329437    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-systemd-units\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.329489 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329479    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-node-log\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.329718 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329498    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/65b590ed-e1a5-467f-9283-7c875dd9ee24-ovnkube-config\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.329718 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329520    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sgbt\" (UniqueName: \"kubernetes.io/projected/8771d72e-12a3-429f-b458-e22298238393-kube-api-access-5sgbt\") pod \"insights-runtime-extractor-2224c\" (UID: \"8771d72e-12a3-429f-b458-e22298238393\") " pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:32.329718 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329567    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttk6z\" (UniqueName: \"kubernetes.io/projected/facedcd2-6b70-4a3b-980c-04dceba2ef0d-kube-api-access-ttk6z\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:37:32.329718 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329597    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-sys\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.329718 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329661    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4545\" (UniqueName: \"kubernetes.io/projected/f926eeff-6488-48c5-9e97-269a6d79dee1-kube-api-access-x4545\") pod \"ingress-canary-mksqw\" (UID: \"f926eeff-6488-48c5-9e97-269a6d79dee1\") " pod="openshift-ingress-canary/ingress-canary-mksqw"
Nov 03 19:37:32.329718 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329688    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-cni-bin\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.329718 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329704    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/65b590ed-e1a5-467f-9283-7c875dd9ee24-ovnkube-config\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.329718 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329714    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8771d72e-12a3-429f-b458-e22298238393-data-volume\") pod \"insights-runtime-extractor-2224c\" (UID: \"8771d72e-12a3-429f-b458-e22298238393\") " pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:32.330034 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329750    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8771d72e-12a3-429f-b458-e22298238393-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2224c\" (UID: \"8771d72e-12a3-429f-b458-e22298238393\") " pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:32.330034 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329784    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-host\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.330034 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329794    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8771d72e-12a3-429f-b458-e22298238393-data-volume\") pod \"insights-runtime-extractor-2224c\" (UID: \"8771d72e-12a3-429f-b458-e22298238393\") " pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:32.330034 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329836    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-hostroot\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.330034 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329882    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-run-openvswitch\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.330034 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329907    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-cni-netd\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.330034 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329923    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/88a27c5e-de31-4090-be4d-6b8d6551e4c7-serviceca\") pod \"node-ca-pswxd\" (UID: \"88a27c5e-de31-4090-be4d-6b8d6551e4c7\") " pod="openshift-image-registry/node-ca-pswxd"
Nov 03 19:37:32.330034 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329931    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8771d72e-12a3-429f-b458-e22298238393-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2224c\" (UID: \"8771d72e-12a3-429f-b458-e22298238393\") " pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:32.330034 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329963    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8771d72e-12a3-429f-b458-e22298238393-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2224c\" (UID: \"8771d72e-12a3-429f-b458-e22298238393\") " pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:32.330034 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.329986    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-kubernetes\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.330034 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330010    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-systemd\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.330519 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330115    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-var-lib-openvswitch\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.330519 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330144    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/65b590ed-e1a5-467f-9283-7c875dd9ee24-env-overrides\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.330519 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330173    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nx7w\" (UniqueName: \"kubernetes.io/projected/88a27c5e-de31-4090-be4d-6b8d6551e4c7-kube-api-access-5nx7w\") pod \"node-ca-pswxd\" (UID: \"88a27c5e-de31-4090-be4d-6b8d6551e4c7\") " pod="openshift-image-registry/node-ca-pswxd"
Nov 03 19:37:32.330519 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330195    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca39ebc9-da71-43c2-893f-c033684c9399-metrics-certs\") pod \"network-metrics-daemon-794nj\" (UID: \"ca39ebc9-da71-43c2-893f-c033684c9399\") " pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:37:32.330519 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330263    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98d32fbe-5011-47a8-b937-725a8cad95a4-original-pull-secret\") pod \"global-pull-secret-syncer-6q5vq\" (UID: \"98d32fbe-5011-47a8-b937-725a8cad95a4\") " pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:37:32.330519 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330284    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8771d72e-12a3-429f-b458-e22298238393-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2224c\" (UID: \"8771d72e-12a3-429f-b458-e22298238393\") " pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:32.330519 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330324    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-sysconfig\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.330519 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330343    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-sysctl-conf\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.330519 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330353    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/65b590ed-e1a5-467f-9283-7c875dd9ee24-env-overrides\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.330519 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330362    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzmv5\" (UniqueName: \"kubernetes.io/projected/b4a42db6-0377-4a5a-94ab-be3f82b332aa-kube-api-access-tzmv5\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:37:32.330519 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330411    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca39ebc9-da71-43c2-893f-c033684c9399-metrics-certs\") pod \"network-metrics-daemon-794nj\" (UID: \"ca39ebc9-da71-43c2-893f-c033684c9399\") " pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:37:32.330519 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330509    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-multus-conf-dir\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.330999 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330539    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-run-multus-certs\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.330999 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330564    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-etc-openvswitch\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.330999 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330591    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/98d32fbe-5011-47a8-b937-725a8cad95a4-original-pull-secret\") pod \"global-pull-secret-syncer-6q5vq\" (UID: \"98d32fbe-5011-47a8-b937-725a8cad95a4\") " pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:37:32.330999 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330611    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-sysctl-d\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.330999 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330643    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f926eeff-6488-48c5-9e97-269a6d79dee1-cert\") pod \"ingress-canary-mksqw\" (UID: \"f926eeff-6488-48c5-9e97-269a6d79dee1\") " pod="openshift-ingress-canary/ingress-canary-mksqw"
Nov 03 19:37:32.330999 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330675    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-cni-binary-copy\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.330999 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330755    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/65b590ed-e1a5-467f-9283-7c875dd9ee24-ovnkube-script-lib\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.330999 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330783    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4a42db6-0377-4a5a-94ab-be3f82b332aa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:37:32.330999 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330806    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/630922bd-4efc-4b44-96d1-fa55bce60e21-hosts-file\") pod \"node-resolver-rzn4t\" (UID: \"630922bd-4efc-4b44-96d1-fa55bce60e21\") " pod="openshift-dns/node-resolver-rzn4t"
Nov 03 19:37:32.330999 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330817    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-cni-binary-copy\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.330999 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330831    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-modprobe-d\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.330999 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330867    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f926eeff-6488-48c5-9e97-269a6d79dee1-cert\") pod \"ingress-canary-mksqw\" (UID: \"f926eeff-6488-48c5-9e97-269a6d79dee1\") " pod="openshift-ingress-canary/ingress-canary-mksqw"
Nov 03 19:37:32.330999 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330869    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-tuned\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.330999 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330915    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-kubelet\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.330999 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330920    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-tuned\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.330999 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330931    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-slash\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.330999 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330956    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/65b590ed-e1a5-467f-9283-7c875dd9ee24-ovnkube-script-lib\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.330999 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.330957    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-run-systemd\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.330999 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.331000    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgdxj\" (UniqueName: \"kubernetes.io/projected/65b590ed-e1a5-467f-9283-7c875dd9ee24-kube-api-access-vgdxj\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.331072    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-var-lib-kubelet\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.331105    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-run-ovn\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.331131    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8771d72e-12a3-429f-b458-e22298238393-crio-socket\") pod \"insights-runtime-extractor-2224c\" (UID: \"8771d72e-12a3-429f-b458-e22298238393\") " pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.331150    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-etc-selinux\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.331164    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-var-lib-kubelet\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.331195    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-run-k8s-cni-cncf-io\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.331254    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-multus-cni-dir\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.331293    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-multus-socket-dir-parent\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.331345    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/65b590ed-e1a5-467f-9283-7c875dd9ee24-ovn-node-metrics-cert\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.331387    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.331461    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-device-dir\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.331499    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng4gr\" (UniqueName: \"kubernetes.io/projected/f951b540-0179-474c-8a2c-7ee804f9d630-kube-api-access-ng4gr\") pod \"network-check-target-pfcnb\" (UID: \"f951b540-0179-474c-8a2c-7ee804f9d630\") " pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.331531    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/65b590ed-e1a5-467f-9283-7c875dd9ee24-ovn-node-metrics-cert\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.331535    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-multus-daemon-config\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.331646    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-log-socket\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.331690    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4a42db6-0377-4a5a-94ab-be3f82b332aa-system-cni-dir\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.331749    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/98d32fbe-5011-47a8-b937-725a8cad95a4-dbus\") pod \"global-pull-secret-syncer-6q5vq\" (UID: \"98d32fbe-5011-47a8-b937-725a8cad95a4\") " pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.331785    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-multus-daemon-config\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.331792    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-run-netns\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.331826    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b1cda986-f386-462d-9956-acc2a4de6e32-agent-certs\") pod \"konnectivity-agent-vh26d\" (UID: \"b1cda986-f386-462d-9956-acc2a4de6e32\") " pod="kube-system/konnectivity-agent-vh26d"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.331859    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b1cda986-f386-462d-9956-acc2a4de6e32-konnectivity-ca\") pod \"konnectivity-agent-vh26d\" (UID: \"b1cda986-f386-462d-9956-acc2a4de6e32\") " pod="kube-system/konnectivity-agent-vh26d"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.331886    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-system-cni-dir\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.331912    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b4a42db6-0377-4a5a-94ab-be3f82b332aa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.331940    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpc4k\" (UniqueName: \"kubernetes.io/projected/ca39ebc9-da71-43c2-893f-c033684c9399-kube-api-access-gpc4k\") pod \"network-metrics-daemon-794nj\" (UID: \"ca39ebc9-da71-43c2-893f-c033684c9399\") " pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.331965    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jktld\" (UniqueName: \"kubernetes.io/projected/630922bd-4efc-4b44-96d1-fa55bce60e21-kube-api-access-jktld\") pod \"node-resolver-rzn4t\" (UID: \"630922bd-4efc-4b44-96d1-fa55bce60e21\") " pod="openshift-dns/node-resolver-rzn4t"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.332012    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zmfm\" (UniqueName: \"kubernetes.io/projected/71c679be-e63f-4ad4-a1b2-5474ec4fa125-kube-api-access-4zmfm\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.332040    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b1cda986-f386-462d-9956-acc2a4de6e32-konnectivity-ca\") pod \"konnectivity-agent-vh26d\" (UID: \"b1cda986-f386-462d-9956-acc2a4de6e32\") " pod="kube-system/konnectivity-agent-vh26d"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.332046    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4a42db6-0377-4a5a-94ab-be3f82b332aa-cnibin\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.332077    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-registration-dir\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.332105    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-sys-fs\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.332127    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b4a42db6-0377-4a5a-94ab-be3f82b332aa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.332129    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/98d32fbe-5011-47a8-b937-725a8cad95a4-kubelet-config\") pod \"global-pull-secret-syncer-6q5vq\" (UID: \"98d32fbe-5011-47a8-b937-725a8cad95a4\") " pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.332169    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71c679be-e63f-4ad4-a1b2-5474ec4fa125-tmp\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.332192    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b4a42db6-0377-4a5a-94ab-be3f82b332aa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.332222    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b1cda986-f386-462d-9956-acc2a4de6e32-agent-certs\") pod \"konnectivity-agent-vh26d\" (UID: \"b1cda986-f386-462d-9956-acc2a4de6e32\") " pod="kube-system/konnectivity-agent-vh26d"
Nov 03 19:37:32.332190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.332234    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-run-netns\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.333451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.332258    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgxw5\" (UniqueName: \"kubernetes.io/projected/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-kube-api-access-dgxw5\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.333451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.332267    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71c679be-e63f-4ad4-a1b2-5474ec4fa125-tmp\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.333451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.332281    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88a27c5e-de31-4090-be4d-6b8d6551e4c7-host\") pod \"node-ca-pswxd\" (UID: \"88a27c5e-de31-4090-be4d-6b8d6551e4c7\") " pod="openshift-image-registry/node-ca-pswxd"
Nov 03 19:37:32.333451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.332384    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4a42db6-0377-4a5a-94ab-be3f82b332aa-cni-binary-copy\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:37:32.333451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.332403    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66pfk\" (UniqueName: \"kubernetes.io/projected/055215b4-9de9-494c-bbff-288cc97baf2f-kube-api-access-66pfk\") pod \"iptables-alerter-htzs4\" (UID: \"055215b4-9de9-494c-bbff-288cc97baf2f\") " pod="openshift-network-operator/iptables-alerter-htzs4"
Nov 03 19:37:32.333451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.332432    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-run\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.333451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.332448    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-var-lib-cni-bin\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.333451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.332462    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-etc-kubernetes\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.333451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.332486    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b4a42db6-0377-4a5a-94ab-be3f82b332aa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:37:32.333451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.332676    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b4a42db6-0377-4a5a-94ab-be3f82b332aa-cni-binary-copy\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:37:32.339965 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.339948    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4545\" (UniqueName: \"kubernetes.io/projected/f926eeff-6488-48c5-9e97-269a6d79dee1-kube-api-access-x4545\") pod \"ingress-canary-mksqw\" (UID: \"f926eeff-6488-48c5-9e97-269a6d79dee1\") " pod="openshift-ingress-canary/ingress-canary-mksqw"
Nov 03 19:37:32.340061 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.340017    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sgbt\" (UniqueName: \"kubernetes.io/projected/8771d72e-12a3-429f-b458-e22298238393-kube-api-access-5sgbt\") pod \"insights-runtime-extractor-2224c\" (UID: \"8771d72e-12a3-429f-b458-e22298238393\") " pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:32.340061 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.340043    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttk6z\" (UniqueName: \"kubernetes.io/projected/facedcd2-6b70-4a3b-980c-04dceba2ef0d-kube-api-access-ttk6z\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:37:32.340272 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.340258    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nx7w\" (UniqueName: \"kubernetes.io/projected/88a27c5e-de31-4090-be4d-6b8d6551e4c7-kube-api-access-5nx7w\") pod \"node-ca-pswxd\" (UID: \"88a27c5e-de31-4090-be4d-6b8d6551e4c7\") " pod="openshift-image-registry/node-ca-pswxd"
Nov 03 19:37:32.340306 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.340269    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzmv5\" (UniqueName: \"kubernetes.io/projected/b4a42db6-0377-4a5a-94ab-be3f82b332aa-kube-api-access-tzmv5\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:37:32.343686 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.343669    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgdxj\" (UniqueName: \"kubernetes.io/projected/65b590ed-e1a5-467f-9283-7c875dd9ee24-kube-api-access-vgdxj\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.344678 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.344656    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng4gr\" (UniqueName: \"kubernetes.io/projected/f951b540-0179-474c-8a2c-7ee804f9d630-kube-api-access-ng4gr\") pod \"network-check-target-pfcnb\" (UID: \"f951b540-0179-474c-8a2c-7ee804f9d630\") " pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:37:32.345512 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.345493    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zmfm\" (UniqueName: \"kubernetes.io/projected/71c679be-e63f-4ad4-a1b2-5474ec4fa125-kube-api-access-4zmfm\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.345663 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.345646    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jktld\" (UniqueName: \"kubernetes.io/projected/630922bd-4efc-4b44-96d1-fa55bce60e21-kube-api-access-jktld\") pod \"node-resolver-rzn4t\" (UID: \"630922bd-4efc-4b44-96d1-fa55bce60e21\") " pod="openshift-dns/node-resolver-rzn4t"
Nov 03 19:37:32.345900 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.345878    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgxw5\" (UniqueName: \"kubernetes.io/projected/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-kube-api-access-dgxw5\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.345967 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.345890    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpc4k\" (UniqueName: \"kubernetes.io/projected/ca39ebc9-da71-43c2-893f-c033684c9399-kube-api-access-gpc4k\") pod \"network-metrics-daemon-794nj\" (UID: \"ca39ebc9-da71-43c2-893f-c033684c9399\") " pod="openshift-multus/network-metrics-daemon-794nj"
Nov 03 19:37:32.346043 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.346023    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66pfk\" (UniqueName: \"kubernetes.io/projected/055215b4-9de9-494c-bbff-288cc97baf2f-kube-api-access-66pfk\") pod \"iptables-alerter-htzs4\" (UID: \"055215b4-9de9-494c-bbff-288cc97baf2f\") " pod="openshift-network-operator/iptables-alerter-htzs4"
Nov 03 19:37:32.406700 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.406632    5591 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:37:32.412174 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.412149    5591 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 19:37:32.416442 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.416416    5591 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 19:37:32.416540 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:37:32.416470    5591 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal"
Nov 03 19:37:32.433005 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.432971    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:32.433110 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433017    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8hqp\" (UniqueName: \"kubernetes.io/projected/300a6502-6932-49fe-936a-ba0c16978c94-kube-api-access-x8hqp\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:32.433110 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433044    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-run-netns\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.433110 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433072    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ac8ac4b-fef2-475d-a40b-9db77cded4d3-tmp-dir\") pod \"dns-default-868lx\" (UID: \"3ac8ac4b-fef2-475d-a40b-9db77cded4d3\") " pod="openshift-dns/dns-default-868lx"
Nov 03 19:37:32.433110 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433098    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4a42db6-0377-4a5a-94ab-be3f82b332aa-cnibin\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:37:32.433403 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433123    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-sys-fs\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:37:32.433403 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433153    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-run-netns\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.433403 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433148    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/98d32fbe-5011-47a8-b937-725a8cad95a4-kubelet-config\") pod \"global-pull-secret-syncer-6q5vq\" (UID: \"98d32fbe-5011-47a8-b937-725a8cad95a4\") " pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:37:32.433403 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433195    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b4a42db6-0377-4a5a-94ab-be3f82b332aa-cnibin\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:37:32.433403 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433232    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-textfile\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:32.433403 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433246    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-sys-fs\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:37:32.433403 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433254    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:32.433403 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433263    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:32.433403 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433267    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ac8ac4b-fef2-475d-a40b-9db77cded4d3-tmp-dir\") pod \"dns-default-868lx\" (UID: \"3ac8ac4b-fef2-475d-a40b-9db77cded4d3\") " pod="openshift-dns/dns-default-868lx"
Nov 03 19:37:32.433403 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433293    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7e50fe5-0dd0-4f59-bc5f-99199500790b-tmp\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:32.433403 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433299    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/98d32fbe-5011-47a8-b937-725a8cad95a4-kubelet-config\") pod \"global-pull-secret-syncer-6q5vq\" (UID: \"98d32fbe-5011-47a8-b937-725a8cad95a4\") " pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:37:32.433403 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433310    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-textfile\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:32.433403 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433318    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-etc-kubernetes\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.433403 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433349    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-etc-kubernetes\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.433403 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433348    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-run-ovn-kubernetes\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.433403 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433363    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:32.433403 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433381    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/300a6502-6932-49fe-936a-ba0c16978c94-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:32.433403 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433390    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-run-ovn-kubernetes\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.433403 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433392    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7e50fe5-0dd0-4f59-bc5f-99199500790b-tmp\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433402    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-cnibin\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433444    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-var-lib-cni-multus\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433468    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-socket-dir\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433467    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/300a6502-6932-49fe-936a-ba0c16978c94-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433489    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-cnibin\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433499    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-var-lib-cni-multus\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433542    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-socket-dir\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433569    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-tls\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433587    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/803910f1-a1f3-4316-ada9-1890e46bbda1-root\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433605    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-sys\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433620    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ac8ac4b-fef2-475d-a40b-9db77cded4d3-metrics-tls\") pod \"dns-default-868lx\" (UID: \"3ac8ac4b-fef2-475d-a40b-9db77cded4d3\") " pod="openshift-dns/dns-default-868lx"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433636    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-web-config\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433684    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-cni-bin\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433716    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-cni-bin\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433747    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-host\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433768    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-sys\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433778    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-host\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433804    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac8ac4b-fef2-475d-a40b-9db77cded4d3-config-volume\") pod \"dns-default-868lx\" (UID: \"3ac8ac4b-fef2-475d-a40b-9db77cded4d3\") " pod="openshift-dns/dns-default-868lx"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433843    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-tls\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433890    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-web-config\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433903    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-hostroot\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433931    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-run-openvswitch\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433949    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3ac8ac4b-fef2-475d-a40b-9db77cded4d3-config-volume\") pod \"dns-default-868lx\" (UID: \"3ac8ac4b-fef2-475d-a40b-9db77cded4d3\") " pod="openshift-dns/dns-default-868lx"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433955    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-cni-netd\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433976    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-hostroot\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433981    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1cd0db81-8027-4c6a-b33c-3b4545351eac-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rvqts\" (UID: \"1cd0db81-8027-4c6a-b33c-3b4545351eac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rvqts"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433994    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-run-openvswitch\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.433999    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-cni-netd\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.434010    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.434010    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ac8ac4b-fef2-475d-a40b-9db77cded4d3-metrics-tls\") pod \"dns-default-868lx\" (UID: \"3ac8ac4b-fef2-475d-a40b-9db77cded4d3\") " pod="openshift-dns/dns-default-868lx"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.434034    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.434090    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-var-lib-openvswitch\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.434122    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/803910f1-a1f3-4316-ada9-1890e46bbda1-metrics-client-ca\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.434146    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6rwg\" (UniqueName: \"kubernetes.io/projected/803910f1-a1f3-4316-ada9-1890e46bbda1-kube-api-access-q6rwg\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.434151    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1cd0db81-8027-4c6a-b33c-3b4545351eac-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rvqts\" (UID: \"1cd0db81-8027-4c6a-b33c-3b4545351eac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rvqts"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.434172    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.434195    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-run-multus-certs\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.434196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.434236    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-sysctl-d\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.435833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.434246    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:32.435833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.434261    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/803910f1-a1f3-4316-ada9-1890e46bbda1-sys\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:32.435833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.434198    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-var-lib-openvswitch\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.435833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.434288    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:32.435833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.434314    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-run-multus-certs\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.435833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.434312    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/630922bd-4efc-4b44-96d1-fa55bce60e21-hosts-file\") pod \"node-resolver-rzn4t\" (UID: \"630922bd-4efc-4b44-96d1-fa55bce60e21\") " pod="openshift-dns/node-resolver-rzn4t"
Nov 03 19:37:32.435833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.434331    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/803910f1-a1f3-4316-ada9-1890e46bbda1-metrics-client-ca\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:32.435833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.434358    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-modprobe-d\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.435833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.434385    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:32.435833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.434437    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/630922bd-4efc-4b44-96d1-fa55bce60e21-hosts-file\") pod \"node-resolver-rzn4t\" (UID: \"630922bd-4efc-4b44-96d1-fa55bce60e21\") " pod="openshift-dns/node-resolver-rzn4t"
Nov 03 19:37:32.435833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.434447    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-sysctl-d\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.435833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.434461    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-config-volume\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:32.435833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.434485    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:32.435833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.434511    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmggx\" (UniqueName: \"kubernetes.io/projected/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-kube-api-access-tmggx\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:32.435833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.434873    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:32.435833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.435138    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-modprobe-d\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.435833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.435263    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-kubelet\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.435833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.435331    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1cd0db81-8027-4c6a-b33c-3b4545351eac-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rvqts\" (UID: \"1cd0db81-8027-4c6a-b33c-3b4545351eac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rvqts"
Nov 03 19:37:32.435833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.435390    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-run-ovn\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.435833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.435418    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:32.435833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.435427    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:32.435833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.435482    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-run-k8s-cni-cncf-io\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.435833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.435509    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-multus-socket-dir-parent\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.435833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.435512    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-kubelet\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.435833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.435639    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:32.435833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.435807    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-config-volume\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:32.436975 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.435883    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-run-k8s-cni-cncf-io\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.436975 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.435948    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-multus-socket-dir-parent\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.436975 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.435965    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1cd0db81-8027-4c6a-b33c-3b4545351eac-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rvqts\" (UID: \"1cd0db81-8027-4c6a-b33c-3b4545351eac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rvqts"
Nov 03 19:37:32.436975 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.435987    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:32.436975 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.436000    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-run-ovn\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.436975 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.436022    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-device-dir\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:37:32.436975 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.436065    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/300a6502-6932-49fe-936a-ba0c16978c94-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:32.436975 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.436129    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-device-dir\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:37:32.436975 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.436255    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-log-socket\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.436975 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.436316    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvddx\" (UniqueName: \"kubernetes.io/projected/1cd0db81-8027-4c6a-b33c-3b4545351eac-kube-api-access-gvddx\") pod \"prometheus-operator-9cd89b74c-rvqts\" (UID: \"1cd0db81-8027-4c6a-b33c-3b4545351eac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rvqts"
Nov 03 19:37:32.436975 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.436367    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:32.436975 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.436375    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/300a6502-6932-49fe-936a-ba0c16978c94-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:32.439366 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.438812    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-system-cni-dir\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.439366 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.438854    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-registration-dir\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:37:32.439366 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.438877    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-run-netns\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.439366 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.438904    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88a27c5e-de31-4090-be4d-6b8d6551e4c7-host\") pod \"node-ca-pswxd\" (UID: \"88a27c5e-de31-4090-be4d-6b8d6551e4c7\") " pod="openshift-image-registry/node-ca-pswxd"
Nov 03 19:37:32.439366 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.438932    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-run\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.439366 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.438961    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f6bc2cf5-3e45-4217-af83-38df92cfee7f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-bnrmk\" (UID: \"f6bc2cf5-3e45-4217-af83-38df92cfee7f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bnrmk"
Nov 03 19:37:32.439366 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.438986    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:32.439366 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439015    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-var-lib-cni-bin\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.439366 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439046    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.439366 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439076    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/055215b4-9de9-494c-bbff-288cc97baf2f-host-slash\") pod \"iptables-alerter-htzs4\" (UID: \"055215b4-9de9-494c-bbff-288cc97baf2f\") " pod="openshift-network-operator/iptables-alerter-htzs4"
Nov 03 19:37:32.439366 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439105    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-lib-modules\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.439366 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439139    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/300a6502-6932-49fe-936a-ba0c16978c94-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:32.439366 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439166    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/300a6502-6932-49fe-936a-ba0c16978c94-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:32.439366 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439316    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-systemd-units\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.439366 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439348    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-node-log\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.440231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439383    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7e50fe5-0dd0-4f59-bc5f-99199500790b-metrics-client-ca\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:32.440231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439413    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-tls\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:32.440231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439439    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-kubernetes\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.440231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439468    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-systemd\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.440231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439498    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:32.440231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439530    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz2hs\" (UniqueName: \"kubernetes.io/projected/f6bc2cf5-3e45-4217-af83-38df92cfee7f-kube-api-access-rz2hs\") pod \"monitoring-plugin-77f4f76594-bnrmk\" (UID: \"f6bc2cf5-3e45-4217-af83-38df92cfee7f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bnrmk"
Nov 03 19:37:32.440231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439554    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-sysconfig\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.440231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439583    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-sysctl-conf\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.440231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439616    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-wtmp\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:32.440231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439647    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/300a6502-6932-49fe-936a-ba0c16978c94-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:32.440231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439677    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-config-out\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:32.440231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439702    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-multus-conf-dir\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.440231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439731    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-etc-openvswitch\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.440231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439769    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1cd0db81-8027-4c6a-b33c-3b4545351eac-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rvqts\" (UID: \"1cd0db81-8027-4c6a-b33c-3b4545351eac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rvqts"
Nov 03 19:37:32.440231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439800    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-accelerators-collector-config\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:32.440231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439835    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4a42db6-0377-4a5a-94ab-be3f82b332aa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:37:32.440231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439866    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:32.440231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439896    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:32.440231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439919    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-slash\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.440231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439946    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-run-systemd\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.440231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.439979    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:32.440231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.440008    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-var-lib-kubelet\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.440231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.440031    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8771d72e-12a3-429f-b458-e22298238393-crio-socket\") pod \"insights-runtime-extractor-2224c\" (UID: \"8771d72e-12a3-429f-b458-e22298238393\") " pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:32.440231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.440218    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-lib-modules\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.440260    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-etc-selinux\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.440291    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-var-lib-kubelet\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.440683    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.440694    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f6bc2cf5-3e45-4217-af83-38df92cfee7f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-bnrmk\" (UID: \"f6bc2cf5-3e45-4217-af83-38df92cfee7f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bnrmk"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.440469    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-run-systemd\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.440514    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-var-lib-cni-bin\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.440708    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-grpc-tls\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.440738    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-tls\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.440380    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-etc-openvswitch\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.440745    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.440780    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n2qg\" (UniqueName: \"kubernetes.io/projected/c7e50fe5-0dd0-4f59-bc5f-99199500790b-kube-api-access-7n2qg\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.440786    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-systemd\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.440810    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-slash\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.440827    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-registration-dir\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.440863    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp6g4\" (UniqueName: \"kubernetes.io/projected/3ac8ac4b-fef2-475d-a40b-9db77cded4d3-kube-api-access-jp6g4\") pod \"dns-default-868lx\" (UID: \"3ac8ac4b-fef2-475d-a40b-9db77cded4d3\") " pod="openshift-dns/dns-default-868lx"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.440904    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-multus-cni-dir\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.440473    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-var-lib-kubelet\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.440936    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.440972    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-sysctl-conf\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441028    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-accelerators-collector-config\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441042    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4a42db6-0377-4a5a-94ab-be3f82b332aa-system-cni-dir\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441093    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b4a42db6-0377-4a5a-94ab-be3f82b332aa-system-cni-dir\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441110    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441130    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b4a42db6-0377-4a5a-94ab-be3f82b332aa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q4tbd\" (UID: \"b4a42db6-0377-4a5a-94ab-be3f82b332aa\") " pod="openshift-multus/multus-additional-cni-plugins-q4tbd"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441152    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-run\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441164    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-config-out\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441169    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-log-socket\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441183    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-run-netns\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441198    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441235    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441244    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-system-cni-dir\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441272    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/98d32fbe-5011-47a8-b937-725a8cad95a4-dbus\") pod \"global-pull-secret-syncer-6q5vq\" (UID: \"98d32fbe-5011-47a8-b937-725a8cad95a4\") " pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441294    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-node-log\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441294    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441327    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/65b590ed-e1a5-467f-9283-7c875dd9ee24-systemd-units\") pod \"ovnkube-node-x88xn\" (UID: \"65b590ed-e1a5-467f-9283-7c875dd9ee24\") " pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:32.441353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441364    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-multus-cni-dir\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.442637 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441391    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c7e50fe5-0dd0-4f59-bc5f-99199500790b-secret-grpc-tls\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:32.442637 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441405    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/055215b4-9de9-494c-bbff-288cc97baf2f-host-slash\") pod \"iptables-alerter-htzs4\" (UID: \"055215b4-9de9-494c-bbff-288cc97baf2f\") " pod="openshift-network-operator/iptables-alerter-htzs4"
Nov 03 19:37:32.442637 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441455    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-host-var-lib-kubelet\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.442637 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441464    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1cd0db81-8027-4c6a-b33c-3b4545351eac-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rvqts\" (UID: \"1cd0db81-8027-4c6a-b33c-3b4545351eac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rvqts"
Nov 03 19:37:32.442637 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441508    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:32.442637 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441510    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/300a6502-6932-49fe-936a-ba0c16978c94-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:32.442637 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441602    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/300a6502-6932-49fe-936a-ba0c16978c94-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:32.442637 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441618    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/32452ae6-08d2-4fb2-863d-72dc09d4c0a2-multus-conf-dir\") pod \"multus-xtsbd\" (UID: \"32452ae6-08d2-4fb2-863d-72dc09d4c0a2\") " pod="openshift-multus/multus-xtsbd"
Nov 03 19:37:32.442637 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441637    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/98d32fbe-5011-47a8-b937-725a8cad95a4-dbus\") pod \"global-pull-secret-syncer-6q5vq\" (UID: \"98d32fbe-5011-47a8-b937-725a8cad95a4\") " pod="kube-system/global-pull-secret-syncer-6q5vq"
Nov 03 19:37:32.442637 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441705    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-sysconfig\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.442637 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441714    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:32.442637 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441783    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/88a27c5e-de31-4090-be4d-6b8d6551e4c7-host\") pod \"node-ca-pswxd\" (UID: \"88a27c5e-de31-4090-be4d-6b8d6551e4c7\") " pod="openshift-image-registry/node-ca-pswxd"
Nov 03 19:37:32.442637 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441787    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/facedcd2-6b70-4a3b-980c-04dceba2ef0d-etc-selinux\") pod \"aws-ebs-csi-driver-node-4h9rf\" (UID: \"facedcd2-6b70-4a3b-980c-04dceba2ef0d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4h9rf"
Nov 03 19:37:32.442637 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441845    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8771d72e-12a3-429f-b458-e22298238393-crio-socket\") pod \"insights-runtime-extractor-2224c\" (UID: \"8771d72e-12a3-429f-b458-e22298238393\") " pod="openshift-insights/insights-runtime-extractor-2224c"
Nov 03 19:37:32.442637 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.441894    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71c679be-e63f-4ad4-a1b2-5474ec4fa125-etc-kubernetes\") pod \"tuned-mgrnm\" (UID: \"71c679be-e63f-4ad4-a1b2-5474ec4fa125\") " pod="openshift-cluster-node-tuning-operator/tuned-mgrnm"
Nov 03 19:37:32.442637 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.442006    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/300a6502-6932-49fe-936a-ba0c16978c94-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:32.442637 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.442074    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7e50fe5-0dd0-4f59-bc5f-99199500790b-metrics-client-ca\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:32.443426 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.443408    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8hqp\" (UniqueName: \"kubernetes.io/projected/300a6502-6932-49fe-936a-ba0c16978c94-kube-api-access-x8hqp\") pod \"kube-state-metrics-5dc8d7d89b-t8nqk\" (UID: \"300a6502-6932-49fe-936a-ba0c16978c94\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-t8nqk"
Nov 03 19:37:32.443820 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.443801    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6rwg\" (UniqueName: \"kubernetes.io/projected/803910f1-a1f3-4316-ada9-1890e46bbda1-kube-api-access-q6rwg\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:32.445499 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.445483    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmggx\" (UniqueName: \"kubernetes.io/projected/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-kube-api-access-tmggx\") pod \"alertmanager-main-0\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:32.450828 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.450806    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz2hs\" (UniqueName: \"kubernetes.io/projected/f6bc2cf5-3e45-4217-af83-38df92cfee7f-kube-api-access-rz2hs\") pod \"monitoring-plugin-77f4f76594-bnrmk\" (UID: \"f6bc2cf5-3e45-4217-af83-38df92cfee7f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bnrmk"
Nov 03 19:37:32.450949 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.450927    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp6g4\" (UniqueName: \"kubernetes.io/projected/3ac8ac4b-fef2-475d-a40b-9db77cded4d3-kube-api-access-jp6g4\") pod \"dns-default-868lx\" (UID: \"3ac8ac4b-fef2-475d-a40b-9db77cded4d3\") " pod="openshift-dns/dns-default-868lx"
Nov 03 19:37:32.450997 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.450974    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n2qg\" (UniqueName: \"kubernetes.io/projected/c7e50fe5-0dd0-4f59-bc5f-99199500790b-kube-api-access-7n2qg\") pod \"thanos-querier-7bfbb78f4c-2gsct\" (UID: \"c7e50fe5-0dd0-4f59-bc5f-99199500790b\") " pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:32.451083 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.451061    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvddx\" (UniqueName: \"kubernetes.io/projected/1cd0db81-8027-4c6a-b33c-3b4545351eac-kube-api-access-gvddx\") pod \"prometheus-operator-9cd89b74c-rvqts\" (UID: \"1cd0db81-8027-4c6a-b33c-3b4545351eac\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rvqts"
Nov 03 19:37:32.542114 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.542078    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/803910f1-a1f3-4316-ada9-1890e46bbda1-root\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:32.542114 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.542123    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/803910f1-a1f3-4316-ada9-1890e46bbda1-sys\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:32.542374 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.542191    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/803910f1-a1f3-4316-ada9-1890e46bbda1-sys\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:32.542374 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.542222    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/803910f1-a1f3-4316-ada9-1890e46bbda1-root\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:32.542374 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.542260    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-wtmp\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:32.542374 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.542355    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/803910f1-a1f3-4316-ada9-1890e46bbda1-node-exporter-wtmp\") pod \"node-exporter-sckcx\" (UID: \"803910f1-a1f3-4316-ada9-1890e46bbda1\") " pod="openshift-monitoring/node-exporter-sckcx"
Nov 03 19:37:32.651608 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:32.651578    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-868lx"
Nov 03 19:37:35.859497 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:35.859466    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7bfbb78f4c-2gsct"
Nov 03 19:37:36.583504 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:36.583475    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bnrmk"
Nov 03 19:37:37.659453 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:37.659418    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vh26d"
Nov 03 19:37:40.597405 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:40.597374    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:41.971450 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:41.971364    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:41.971789 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:41.971606    5591 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 19:37:42.284396 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:42.284373    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:37:44.970914 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:44.970878    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x88xn"
Nov 03 19:37:53.651750 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:53.651715    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:37:53.652262 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:53.652099    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerName="alertmanager" containerID="cri-o://e281919b8c165d59b742e4aaa3f9e48d836a4437355c03c4ab167cd60bb360bd" gracePeriod=120
Nov 03 19:37:53.652262 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:53.652105    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerName="kube-rbac-proxy" containerID="cri-o://987605c633836498a8cf1f0041052cb35897382152a89641b4af7f70e626d264" gracePeriod=120
Nov 03 19:37:53.652262 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:53.652152    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerName="prom-label-proxy" containerID="cri-o://28bba717aa2ceae16309031ea60bbadef31e99a8ffde5698d7ebeec24e02d467" gracePeriod=120
Nov 03 19:37:53.652262 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:53.652104    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerName="kube-rbac-proxy-metric" containerID="cri-o://89417e2a306c84e55d86c07a221fb764a986678e3d257bd6c5c7a83d0360df8e" gracePeriod=120
Nov 03 19:37:53.652262 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:53.652222    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerName="kube-rbac-proxy-web" containerID="cri-o://fd06f1ac57f7a0e0cf584f9d40064856fe743086cbf9b2b002655cc6c3d62aad" gracePeriod=120
Nov 03 19:37:53.652509 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:53.652302    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerName="config-reloader" containerID="cri-o://a1f1af32f35471807dd91f0988f71f6fcbedd923261adceee04ba26995263dc6" gracePeriod=120
Nov 03 19:37:54.488245 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:54.488215    5591 generic.go:356] "Generic (PLEG): container finished" podID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerID="28bba717aa2ceae16309031ea60bbadef31e99a8ffde5698d7ebeec24e02d467" exitCode=0
Nov 03 19:37:54.488245 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:54.488239    5591 generic.go:356] "Generic (PLEG): container finished" podID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerID="987605c633836498a8cf1f0041052cb35897382152a89641b4af7f70e626d264" exitCode=0
Nov 03 19:37:54.488245 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:54.488246    5591 generic.go:356] "Generic (PLEG): container finished" podID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerID="a1f1af32f35471807dd91f0988f71f6fcbedd923261adceee04ba26995263dc6" exitCode=0
Nov 03 19:37:54.488245 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:54.488254    5591 generic.go:356] "Generic (PLEG): container finished" podID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerID="e281919b8c165d59b742e4aaa3f9e48d836a4437355c03c4ab167cd60bb360bd" exitCode=0
Nov 03 19:37:54.488491 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:54.488235    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b9ebd2a-44e9-41f6-8bef-c41257c9a163","Type":"ContainerDied","Data":"28bba717aa2ceae16309031ea60bbadef31e99a8ffde5698d7ebeec24e02d467"}
Nov 03 19:37:54.488491 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:54.488363    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b9ebd2a-44e9-41f6-8bef-c41257c9a163","Type":"ContainerDied","Data":"987605c633836498a8cf1f0041052cb35897382152a89641b4af7f70e626d264"}
Nov 03 19:37:54.488491 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:54.488392    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b9ebd2a-44e9-41f6-8bef-c41257c9a163","Type":"ContainerDied","Data":"a1f1af32f35471807dd91f0988f71f6fcbedd923261adceee04ba26995263dc6"}
Nov 03 19:37:54.488491 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:54.488415    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b9ebd2a-44e9-41f6-8bef-c41257c9a163","Type":"ContainerDied","Data":"e281919b8c165d59b742e4aaa3f9e48d836a4437355c03c4ab167cd60bb360bd"}
Nov 03 19:37:54.887281 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:54.887259    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.014411 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.014384    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-web-config\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-web-config\") pod \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") "
Nov 03 19:37:55.014573 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.014428    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-tls-assets\" (UniqueName: \"kubernetes.io/projected/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-tls-assets\") pod \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") "
Nov 03 19:37:55.014573 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.014458    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-config-out\") pod \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") "
Nov 03 19:37:55.014573 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.014476    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-config-volume\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-config-volume\") pod \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") "
Nov 03 19:37:55.014573 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.014510    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-kube-api-access-tmggx\" (UniqueName: \"kubernetes.io/projected/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-kube-api-access-tmggx\") pod \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") "
Nov 03 19:37:55.014573 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.014536    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy-metric\") pod \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") "
Nov 03 19:37:55.014573 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.014555    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-alertmanager-main-db\") pod \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") "
Nov 03 19:37:55.014877 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.014575    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy-web\") pod \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") "
Nov 03 19:37:55.015061 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.015024    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-alertmanager-main-db" pod "8b9ebd2a-44e9-41f6-8bef-c41257c9a163" (UID: "8b9ebd2a-44e9-41f6-8bef-c41257c9a163"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:37:55.015963 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.015310    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-main-tls\") pod \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") "
Nov 03 19:37:55.015963 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.015370    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-cluster-tls-config\") pod \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") "
Nov 03 19:37:55.015963 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.015410    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy\") pod \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") "
Nov 03 19:37:55.015963 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.015445    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-alertmanager-trusted-ca-bundle\") pod \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") "
Nov 03 19:37:55.015963 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.015507    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-metrics-client-ca\") pod \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\" (UID: \"8b9ebd2a-44e9-41f6-8bef-c41257c9a163\") "
Nov 03 19:37:55.015963 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.015737    5591 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-alertmanager-main-db\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:55.017466 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.017273    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-metrics-client-ca" pod "8b9ebd2a-44e9-41f6-8bef-c41257c9a163" (UID: "8b9ebd2a-44e9-41f6-8bef-c41257c9a163"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:55.017773 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.017745    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-alertmanager-trusted-ca-bundle" pod "8b9ebd2a-44e9-41f6-8bef-c41257c9a163" (UID: "8b9ebd2a-44e9-41f6-8bef-c41257c9a163"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:55.018556 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.018520    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-config-out" pod "8b9ebd2a-44e9-41f6-8bef-c41257c9a163" (UID: "8b9ebd2a-44e9-41f6-8bef-c41257c9a163"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:37:55.018659 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.018633    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-tls-assets" pod "8b9ebd2a-44e9-41f6-8bef-c41257c9a163" (UID: "8b9ebd2a-44e9-41f6-8bef-c41257c9a163"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:37:55.018849 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.018826    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy-metric" pod "8b9ebd2a-44e9-41f6-8bef-c41257c9a163" (UID: "8b9ebd2a-44e9-41f6-8bef-c41257c9a163"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:37:55.018947 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.018888    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-kube-api-access-tmggx" pod "8b9ebd2a-44e9-41f6-8bef-c41257c9a163" (UID: "8b9ebd2a-44e9-41f6-8bef-c41257c9a163"). InnerVolumeSpecName "kube-api-access-tmggx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:37:55.019023 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.018991    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-config-volume" pod "8b9ebd2a-44e9-41f6-8bef-c41257c9a163" (UID: "8b9ebd2a-44e9-41f6-8bef-c41257c9a163"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:37:55.019192 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.019169    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy-web" pod "8b9ebd2a-44e9-41f6-8bef-c41257c9a163" (UID: "8b9ebd2a-44e9-41f6-8bef-c41257c9a163"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:37:55.019339 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.019319    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-main-tls" pod "8b9ebd2a-44e9-41f6-8bef-c41257c9a163" (UID: "8b9ebd2a-44e9-41f6-8bef-c41257c9a163"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:37:55.019841 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.019822    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy" pod "8b9ebd2a-44e9-41f6-8bef-c41257c9a163" (UID: "8b9ebd2a-44e9-41f6-8bef-c41257c9a163"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:37:55.023406 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.023384    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-cluster-tls-config" pod "8b9ebd2a-44e9-41f6-8bef-c41257c9a163" (UID: "8b9ebd2a-44e9-41f6-8bef-c41257c9a163"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:37:55.028899 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.028877    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-web-config" pod "8b9ebd2a-44e9-41f6-8bef-c41257c9a163" (UID: "8b9ebd2a-44e9-41f6-8bef-c41257c9a163"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:37:55.116385 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.116354    5591 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-main-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:55.116385 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.116383    5591 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-cluster-tls-config\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:55.116575 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.116396    5591 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:55.116575 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.116405    5591 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:55.116575 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.116414    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-metrics-client-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:55.116575 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.116447    5591 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-web-config\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:55.116575 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.116455    5591 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-tls-assets\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:55.116575 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.116464    5591 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-config-out\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:55.116575 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.116472    5591 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:55.116575 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.116481    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmggx\" (UniqueName: \"kubernetes.io/projected/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-kube-api-access-tmggx\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:55.116575 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.116490    5591 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:55.116575 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.116499    5591 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b9ebd2a-44e9-41f6-8bef-c41257c9a163-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:55.496126 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.496092    5591 generic.go:356] "Generic (PLEG): container finished" podID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerID="89417e2a306c84e55d86c07a221fb764a986678e3d257bd6c5c7a83d0360df8e" exitCode=0
Nov 03 19:37:55.496126 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.496118    5591 generic.go:356] "Generic (PLEG): container finished" podID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerID="fd06f1ac57f7a0e0cf584f9d40064856fe743086cbf9b2b002655cc6c3d62aad" exitCode=0
Nov 03 19:37:55.496344 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.496170    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.496344 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.496168    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b9ebd2a-44e9-41f6-8bef-c41257c9a163","Type":"ContainerDied","Data":"89417e2a306c84e55d86c07a221fb764a986678e3d257bd6c5c7a83d0360df8e"}
Nov 03 19:37:55.496344 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.496253    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b9ebd2a-44e9-41f6-8bef-c41257c9a163","Type":"ContainerDied","Data":"fd06f1ac57f7a0e0cf584f9d40064856fe743086cbf9b2b002655cc6c3d62aad"}
Nov 03 19:37:55.496344 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.496290    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8b9ebd2a-44e9-41f6-8bef-c41257c9a163","Type":"ContainerDied","Data":"77de5d1cd3a6ba081b6080676f08c8275491705c308f21e922697ae3ed8f06dc"}
Nov 03 19:37:55.496344 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.496319    5591 scope.go:122] "RemoveContainer" containerID="28bba717aa2ceae16309031ea60bbadef31e99a8ffde5698d7ebeec24e02d467"
Nov 03 19:37:55.504934 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.504917    5591 scope.go:122] "RemoveContainer" containerID="89417e2a306c84e55d86c07a221fb764a986678e3d257bd6c5c7a83d0360df8e"
Nov 03 19:37:55.512059 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.512037    5591 scope.go:122] "RemoveContainer" containerID="987605c633836498a8cf1f0041052cb35897382152a89641b4af7f70e626d264"
Nov 03 19:37:55.518734 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.518715    5591 scope.go:122] "RemoveContainer" containerID="fd06f1ac57f7a0e0cf584f9d40064856fe743086cbf9b2b002655cc6c3d62aad"
Nov 03 19:37:55.520476 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.520460    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:37:55.524290 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.524244    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:37:55.526476 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.526456    5591 scope.go:122] "RemoveContainer" containerID="a1f1af32f35471807dd91f0988f71f6fcbedd923261adceee04ba26995263dc6"
Nov 03 19:37:55.532937 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.532922    5591 scope.go:122] "RemoveContainer" containerID="e281919b8c165d59b742e4aaa3f9e48d836a4437355c03c4ab167cd60bb360bd"
Nov 03 19:37:55.539581 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.539564    5591 scope.go:122] "RemoveContainer" containerID="5d900c564cd425ff4cebb7144fdc0d60bf5953d447cc834619b884d13dce2b77"
Nov 03 19:37:55.546004 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.545983    5591 scope.go:122] "RemoveContainer" containerID="28bba717aa2ceae16309031ea60bbadef31e99a8ffde5698d7ebeec24e02d467"
Nov 03 19:37:55.546339 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:37:55.546315    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28bba717aa2ceae16309031ea60bbadef31e99a8ffde5698d7ebeec24e02d467\": container with ID starting with 28bba717aa2ceae16309031ea60bbadef31e99a8ffde5698d7ebeec24e02d467 not found: ID does not exist" containerID="28bba717aa2ceae16309031ea60bbadef31e99a8ffde5698d7ebeec24e02d467"
Nov 03 19:37:55.546539 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.546347    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28bba717aa2ceae16309031ea60bbadef31e99a8ffde5698d7ebeec24e02d467"} err="failed to get container status \"28bba717aa2ceae16309031ea60bbadef31e99a8ffde5698d7ebeec24e02d467\": rpc error: code = NotFound desc = could not find container \"28bba717aa2ceae16309031ea60bbadef31e99a8ffde5698d7ebeec24e02d467\": container with ID starting with 28bba717aa2ceae16309031ea60bbadef31e99a8ffde5698d7ebeec24e02d467 not found: ID does not exist"
Nov 03 19:37:55.546539 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.546384    5591 scope.go:122] "RemoveContainer" containerID="89417e2a306c84e55d86c07a221fb764a986678e3d257bd6c5c7a83d0360df8e"
Nov 03 19:37:55.546666 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:37:55.546628    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89417e2a306c84e55d86c07a221fb764a986678e3d257bd6c5c7a83d0360df8e\": container with ID starting with 89417e2a306c84e55d86c07a221fb764a986678e3d257bd6c5c7a83d0360df8e not found: ID does not exist" containerID="89417e2a306c84e55d86c07a221fb764a986678e3d257bd6c5c7a83d0360df8e"
Nov 03 19:37:55.546666 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.546648    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89417e2a306c84e55d86c07a221fb764a986678e3d257bd6c5c7a83d0360df8e"} err="failed to get container status \"89417e2a306c84e55d86c07a221fb764a986678e3d257bd6c5c7a83d0360df8e\": rpc error: code = NotFound desc = could not find container \"89417e2a306c84e55d86c07a221fb764a986678e3d257bd6c5c7a83d0360df8e\": container with ID starting with 89417e2a306c84e55d86c07a221fb764a986678e3d257bd6c5c7a83d0360df8e not found: ID does not exist"
Nov 03 19:37:55.546666 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.546661    5591 scope.go:122] "RemoveContainer" containerID="987605c633836498a8cf1f0041052cb35897382152a89641b4af7f70e626d264"
Nov 03 19:37:55.546923 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:37:55.546905    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"987605c633836498a8cf1f0041052cb35897382152a89641b4af7f70e626d264\": container with ID starting with 987605c633836498a8cf1f0041052cb35897382152a89641b4af7f70e626d264 not found: ID does not exist" containerID="987605c633836498a8cf1f0041052cb35897382152a89641b4af7f70e626d264"
Nov 03 19:37:55.546968 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.546929    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"987605c633836498a8cf1f0041052cb35897382152a89641b4af7f70e626d264"} err="failed to get container status \"987605c633836498a8cf1f0041052cb35897382152a89641b4af7f70e626d264\": rpc error: code = NotFound desc = could not find container \"987605c633836498a8cf1f0041052cb35897382152a89641b4af7f70e626d264\": container with ID starting with 987605c633836498a8cf1f0041052cb35897382152a89641b4af7f70e626d264 not found: ID does not exist"
Nov 03 19:37:55.546968 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.546944    5591 scope.go:122] "RemoveContainer" containerID="fd06f1ac57f7a0e0cf584f9d40064856fe743086cbf9b2b002655cc6c3d62aad"
Nov 03 19:37:55.547190 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:37:55.547172    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd06f1ac57f7a0e0cf584f9d40064856fe743086cbf9b2b002655cc6c3d62aad\": container with ID starting with fd06f1ac57f7a0e0cf584f9d40064856fe743086cbf9b2b002655cc6c3d62aad not found: ID does not exist" containerID="fd06f1ac57f7a0e0cf584f9d40064856fe743086cbf9b2b002655cc6c3d62aad"
Nov 03 19:37:55.547493 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.547472    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd06f1ac57f7a0e0cf584f9d40064856fe743086cbf9b2b002655cc6c3d62aad"} err="failed to get container status \"fd06f1ac57f7a0e0cf584f9d40064856fe743086cbf9b2b002655cc6c3d62aad\": rpc error: code = NotFound desc = could not find container \"fd06f1ac57f7a0e0cf584f9d40064856fe743086cbf9b2b002655cc6c3d62aad\": container with ID starting with fd06f1ac57f7a0e0cf584f9d40064856fe743086cbf9b2b002655cc6c3d62aad not found: ID does not exist"
Nov 03 19:37:55.547567 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.547498    5591 scope.go:122] "RemoveContainer" containerID="a1f1af32f35471807dd91f0988f71f6fcbedd923261adceee04ba26995263dc6"
Nov 03 19:37:55.547760 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:37:55.547744    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f1af32f35471807dd91f0988f71f6fcbedd923261adceee04ba26995263dc6\": container with ID starting with a1f1af32f35471807dd91f0988f71f6fcbedd923261adceee04ba26995263dc6 not found: ID does not exist" containerID="a1f1af32f35471807dd91f0988f71f6fcbedd923261adceee04ba26995263dc6"
Nov 03 19:37:55.547803 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.547770    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f1af32f35471807dd91f0988f71f6fcbedd923261adceee04ba26995263dc6"} err="failed to get container status \"a1f1af32f35471807dd91f0988f71f6fcbedd923261adceee04ba26995263dc6\": rpc error: code = NotFound desc = could not find container \"a1f1af32f35471807dd91f0988f71f6fcbedd923261adceee04ba26995263dc6\": container with ID starting with a1f1af32f35471807dd91f0988f71f6fcbedd923261adceee04ba26995263dc6 not found: ID does not exist"
Nov 03 19:37:55.547803 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.547788    5591 scope.go:122] "RemoveContainer" containerID="e281919b8c165d59b742e4aaa3f9e48d836a4437355c03c4ab167cd60bb360bd"
Nov 03 19:37:55.547985 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:37:55.547969    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e281919b8c165d59b742e4aaa3f9e48d836a4437355c03c4ab167cd60bb360bd\": container with ID starting with e281919b8c165d59b742e4aaa3f9e48d836a4437355c03c4ab167cd60bb360bd not found: ID does not exist" containerID="e281919b8c165d59b742e4aaa3f9e48d836a4437355c03c4ab167cd60bb360bd"
Nov 03 19:37:55.548036 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.547989    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e281919b8c165d59b742e4aaa3f9e48d836a4437355c03c4ab167cd60bb360bd"} err="failed to get container status \"e281919b8c165d59b742e4aaa3f9e48d836a4437355c03c4ab167cd60bb360bd\": rpc error: code = NotFound desc = could not find container \"e281919b8c165d59b742e4aaa3f9e48d836a4437355c03c4ab167cd60bb360bd\": container with ID starting with e281919b8c165d59b742e4aaa3f9e48d836a4437355c03c4ab167cd60bb360bd not found: ID does not exist"
Nov 03 19:37:55.548036 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.548000    5591 scope.go:122] "RemoveContainer" containerID="5d900c564cd425ff4cebb7144fdc0d60bf5953d447cc834619b884d13dce2b77"
Nov 03 19:37:55.548232 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:37:55.548197    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d900c564cd425ff4cebb7144fdc0d60bf5953d447cc834619b884d13dce2b77\": container with ID starting with 5d900c564cd425ff4cebb7144fdc0d60bf5953d447cc834619b884d13dce2b77 not found: ID does not exist" containerID="5d900c564cd425ff4cebb7144fdc0d60bf5953d447cc834619b884d13dce2b77"
Nov 03 19:37:55.548281 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.548235    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d900c564cd425ff4cebb7144fdc0d60bf5953d447cc834619b884d13dce2b77"} err="failed to get container status \"5d900c564cd425ff4cebb7144fdc0d60bf5953d447cc834619b884d13dce2b77\": rpc error: code = NotFound desc = could not find container \"5d900c564cd425ff4cebb7144fdc0d60bf5953d447cc834619b884d13dce2b77\": container with ID starting with 5d900c564cd425ff4cebb7144fdc0d60bf5953d447cc834619b884d13dce2b77 not found: ID does not exist"
Nov 03 19:37:55.548281 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.548248    5591 scope.go:122] "RemoveContainer" containerID="28bba717aa2ceae16309031ea60bbadef31e99a8ffde5698d7ebeec24e02d467"
Nov 03 19:37:55.548457 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.548439    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28bba717aa2ceae16309031ea60bbadef31e99a8ffde5698d7ebeec24e02d467"} err="failed to get container status \"28bba717aa2ceae16309031ea60bbadef31e99a8ffde5698d7ebeec24e02d467\": rpc error: code = NotFound desc = could not find container \"28bba717aa2ceae16309031ea60bbadef31e99a8ffde5698d7ebeec24e02d467\": container with ID starting with 28bba717aa2ceae16309031ea60bbadef31e99a8ffde5698d7ebeec24e02d467 not found: ID does not exist"
Nov 03 19:37:55.548521 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.548459    5591 scope.go:122] "RemoveContainer" containerID="89417e2a306c84e55d86c07a221fb764a986678e3d257bd6c5c7a83d0360df8e"
Nov 03 19:37:55.548730 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.548712    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89417e2a306c84e55d86c07a221fb764a986678e3d257bd6c5c7a83d0360df8e"} err="failed to get container status \"89417e2a306c84e55d86c07a221fb764a986678e3d257bd6c5c7a83d0360df8e\": rpc error: code = NotFound desc = could not find container \"89417e2a306c84e55d86c07a221fb764a986678e3d257bd6c5c7a83d0360df8e\": container with ID starting with 89417e2a306c84e55d86c07a221fb764a986678e3d257bd6c5c7a83d0360df8e not found: ID does not exist"
Nov 03 19:37:55.548777 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.548733    5591 scope.go:122] "RemoveContainer" containerID="987605c633836498a8cf1f0041052cb35897382152a89641b4af7f70e626d264"
Nov 03 19:37:55.548950 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.548933    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"987605c633836498a8cf1f0041052cb35897382152a89641b4af7f70e626d264"} err="failed to get container status \"987605c633836498a8cf1f0041052cb35897382152a89641b4af7f70e626d264\": rpc error: code = NotFound desc = could not find container \"987605c633836498a8cf1f0041052cb35897382152a89641b4af7f70e626d264\": container with ID starting with 987605c633836498a8cf1f0041052cb35897382152a89641b4af7f70e626d264 not found: ID does not exist"
Nov 03 19:37:55.548994 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.548950    5591 scope.go:122] "RemoveContainer" containerID="fd06f1ac57f7a0e0cf584f9d40064856fe743086cbf9b2b002655cc6c3d62aad"
Nov 03 19:37:55.549166 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.549147    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd06f1ac57f7a0e0cf584f9d40064856fe743086cbf9b2b002655cc6c3d62aad"} err="failed to get container status \"fd06f1ac57f7a0e0cf584f9d40064856fe743086cbf9b2b002655cc6c3d62aad\": rpc error: code = NotFound desc = could not find container \"fd06f1ac57f7a0e0cf584f9d40064856fe743086cbf9b2b002655cc6c3d62aad\": container with ID starting with fd06f1ac57f7a0e0cf584f9d40064856fe743086cbf9b2b002655cc6c3d62aad not found: ID does not exist"
Nov 03 19:37:55.549242 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.549167    5591 scope.go:122] "RemoveContainer" containerID="a1f1af32f35471807dd91f0988f71f6fcbedd923261adceee04ba26995263dc6"
Nov 03 19:37:55.549422 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.549402    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f1af32f35471807dd91f0988f71f6fcbedd923261adceee04ba26995263dc6"} err="failed to get container status \"a1f1af32f35471807dd91f0988f71f6fcbedd923261adceee04ba26995263dc6\": rpc error: code = NotFound desc = could not find container \"a1f1af32f35471807dd91f0988f71f6fcbedd923261adceee04ba26995263dc6\": container with ID starting with a1f1af32f35471807dd91f0988f71f6fcbedd923261adceee04ba26995263dc6 not found: ID does not exist"
Nov 03 19:37:55.549422 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.549419    5591 scope.go:122] "RemoveContainer" containerID="e281919b8c165d59b742e4aaa3f9e48d836a4437355c03c4ab167cd60bb360bd"
Nov 03 19:37:55.549623 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.549606    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e281919b8c165d59b742e4aaa3f9e48d836a4437355c03c4ab167cd60bb360bd"} err="failed to get container status \"e281919b8c165d59b742e4aaa3f9e48d836a4437355c03c4ab167cd60bb360bd\": rpc error: code = NotFound desc = could not find container \"e281919b8c165d59b742e4aaa3f9e48d836a4437355c03c4ab167cd60bb360bd\": container with ID starting with e281919b8c165d59b742e4aaa3f9e48d836a4437355c03c4ab167cd60bb360bd not found: ID does not exist"
Nov 03 19:37:55.549674 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.549623    5591 scope.go:122] "RemoveContainer" containerID="5d900c564cd425ff4cebb7144fdc0d60bf5953d447cc834619b884d13dce2b77"
Nov 03 19:37:55.549863 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.549845    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d900c564cd425ff4cebb7144fdc0d60bf5953d447cc834619b884d13dce2b77"} err="failed to get container status \"5d900c564cd425ff4cebb7144fdc0d60bf5953d447cc834619b884d13dce2b77\": rpc error: code = NotFound desc = could not find container \"5d900c564cd425ff4cebb7144fdc0d60bf5953d447cc834619b884d13dce2b77\": container with ID starting with 5d900c564cd425ff4cebb7144fdc0d60bf5953d447cc834619b884d13dce2b77 not found: ID does not exist"
Nov 03 19:37:55.558103 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.558070    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:37:55.558403 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.558389    5591 cpu_manager.go:414] "containerMap: removing container" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerName="kube-rbac-proxy-metric"
Nov 03 19:37:55.558450 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.558406    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerName="kube-rbac-proxy-metric"
Nov 03 19:37:55.558450 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.558414    5591 cpu_manager.go:414] "containerMap: removing container" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerName="kube-rbac-proxy-web"
Nov 03 19:37:55.558450 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.558420    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerName="kube-rbac-proxy-web"
Nov 03 19:37:55.558450 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.558433    5591 cpu_manager.go:414] "containerMap: removing container" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerName="init-config-reloader"
Nov 03 19:37:55.558450 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.558437    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerName="init-config-reloader"
Nov 03 19:37:55.558450 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.558445    5591 cpu_manager.go:414] "containerMap: removing container" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerName="kube-rbac-proxy"
Nov 03 19:37:55.558450 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.558449    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerName="kube-rbac-proxy"
Nov 03 19:37:55.558645 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.558464    5591 cpu_manager.go:414] "containerMap: removing container" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerName="config-reloader"
Nov 03 19:37:55.558645 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.558468    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerName="config-reloader"
Nov 03 19:37:55.558645 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.558477    5591 cpu_manager.go:414] "containerMap: removing container" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerName="prom-label-proxy"
Nov 03 19:37:55.558645 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.558481    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerName="prom-label-proxy"
Nov 03 19:37:55.558645 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.558488    5591 cpu_manager.go:414] "containerMap: removing container" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerName="alertmanager"
Nov 03 19:37:55.558645 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.558493    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerName="alertmanager"
Nov 03 19:37:55.558645 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.558528    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerName="kube-rbac-proxy-metric"
Nov 03 19:37:55.558645 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.558535    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerName="kube-rbac-proxy-web"
Nov 03 19:37:55.558645 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.558541    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerName="init-config-reloader"
Nov 03 19:37:55.558645 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.558546    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerName="kube-rbac-proxy"
Nov 03 19:37:55.558645 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.558552    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerName="config-reloader"
Nov 03 19:37:55.558645 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.558557    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerName="prom-label-proxy"
Nov 03 19:37:55.558645 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.558562    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" containerName="alertmanager"
Nov 03 19:37:55.561889 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.561876    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.566474 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.566446    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 19:37:55.566474 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.566467    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 19:37:55.566717 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.566477    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 19:37:55.566717 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.566571    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 19:37:55.566717 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.566618    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 19:37:55.566717 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.566631    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 19:37:55.566717 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.566639    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 19:37:55.566717 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.566634    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 19:37:55.566917 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.566756    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pz944\""
Nov 03 19:37:55.569360 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.568799    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 19:37:55.579987 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.579968    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:37:55.721297 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.721265    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.721461 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.721303    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.721461 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.721339    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.721558 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.721479    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-config-volume\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.721558 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.721511    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-config-out\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.721558 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.721526    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.721558 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.721550    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxch6\" (UniqueName: \"kubernetes.io/projected/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-kube-api-access-jxch6\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.721691 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.721640    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.721691 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.721662    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-web-config\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.721691 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.721680    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.721791 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.721695    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.721791 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.721710    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.721791 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.721743    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.822912 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.822871    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-config-out\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.822912 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.822909    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.823150 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.822941    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxch6\" (UniqueName: \"kubernetes.io/projected/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-kube-api-access-jxch6\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.823150 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.822972    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.823150 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.822999    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-web-config\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.823150 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.823026    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.823150 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.823047    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.823150 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.823067    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.823150 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.823090    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.823150 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.823135    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.823571 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.823163    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.823571 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.823237    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.823571 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.823278    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-config-volume\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.823571 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.823317    5591 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 19:37:55.824325 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.823882    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.824591 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.824534    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.824816 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.824794    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.826448 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.826294    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-config-out\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.826448 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.826391    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.826448 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.826411    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-config-volume\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.826671 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.826538    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.826832 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.826807    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.827150 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.827134    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-web-config\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.827379 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.827353    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.827504 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.827486    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.828440 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.828425    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.835231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.835192    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxch6\" (UniqueName: \"kubernetes.io/projected/a2dda2e6-5ea0-4469-9cf1-403560cfc3cb-kube-api-access-jxch6\") pod \"alertmanager-main-0\" (UID: \"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:55.878458 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:55.878431    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 19:37:56.161177 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:56.161144    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 19:37:56.164328 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:37:56.164297    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2dda2e6_5ea0_4469_9cf1_403560cfc3cb.slice/crio-c45fa0893877513f5b2d19c4659f787f32593e201db5f80ddfa9a9565406635e WatchSource:0}: Error finding container c45fa0893877513f5b2d19c4659f787f32593e201db5f80ddfa9a9565406635e: Status 404 returned error can't find the container with id c45fa0893877513f5b2d19c4659f787f32593e201db5f80ddfa9a9565406635e
Nov 03 19:37:56.502762 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:56.502681    5591 generic.go:356] "Generic (PLEG): container finished" podID="a2dda2e6-5ea0-4469-9cf1-403560cfc3cb" containerID="fabdf049fca206de2bc6509bc71bdb171ce91bc524d3e99461ac1464250e73c7" exitCode=0
Nov 03 19:37:56.502762 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:56.502723    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb","Type":"ContainerDied","Data":"fabdf049fca206de2bc6509bc71bdb171ce91bc524d3e99461ac1464250e73c7"}
Nov 03 19:37:56.502930 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:56.502772    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb","Type":"ContainerStarted","Data":"c45fa0893877513f5b2d19c4659f787f32593e201db5f80ddfa9a9565406635e"}
Nov 03 19:37:57.346284 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:57.346253    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b9ebd2a-44e9-41f6-8bef-c41257c9a163" path="/var/lib/kubelet/pods/8b9ebd2a-44e9-41f6-8bef-c41257c9a163/volumes"
Nov 03 19:37:57.509441 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:57.509406    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb","Type":"ContainerStarted","Data":"dcbeddb72807db3e59d08c7279ed2668925e30d60394fa4aceef18d73237f907"}
Nov 03 19:37:57.509609 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:57.509470    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb","Type":"ContainerStarted","Data":"d7de0484cbe38e21ffd1e34ca79ff163a42a50a3f36926003168da8e859d0969"}
Nov 03 19:37:57.509609 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:57.509498    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb","Type":"ContainerStarted","Data":"28237655ff60f3eb7c41ce139269d6d0a13e17d01475f9718e9ace590558ba3d"}
Nov 03 19:37:57.509609 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:57.509519    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb","Type":"ContainerStarted","Data":"418c099aba32ecd5c2d7aa8dc7aefaddd20ba4e97378a45943d20da966b6adb5"}
Nov 03 19:37:57.509609 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:57.509547    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb","Type":"ContainerStarted","Data":"b5856dca59253231becb51992cc8e508820c400639ff651c2caac0784b54eaf9"}
Nov 03 19:37:57.509609 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:57.509574    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2dda2e6-5ea0-4469-9cf1-403560cfc3cb","Type":"ContainerStarted","Data":"3b80defc2c1601d570ed65cb4dc8a332b1d688bde8f5f160c73d71b64b1e8e11"}
Nov 03 19:37:57.544297 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:37:57.544145    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.53366876 podStartE2EDuration="2.544126732s" totalImagesPullingTime="0s" totalInitContainerRuntime="10.457972ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:37:57.542956796 +0000 UTC m=+26.315249806" watchObservedRunningTime="2026-11-03 19:37:57.544126732 +0000 UTC m=+26.316419752"
Nov 03 19:37:58.722019 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:37:58.721983    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:38:02.286704 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:38:02.286677    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pfcnb"
Nov 03 19:38:28.722644 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:38:28.722600    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:38:58.722389 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:38:58.722351    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:39:28.721672 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:39:28.721638    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:39:58.721957 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:39:58.721914    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:40:28.721542 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:40:28.721502    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:40:58.721906 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:40:58.721869    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:41:16.925756 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:16.925726    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd"]
Nov 03 19:41:16.929072 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:16.929049    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd"
Nov 03 19:41:16.934350 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:16.934331    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-6t477\""
Nov 03 19:41:16.934547 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:16.934526    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 19:41:16.934605 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:16.934575    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 19:41:16.994261 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:16.994225    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd"]
Nov 03 19:41:17.023198 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.023163    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09418682-d550-4d4c-bf98-0b2e6fad1d1c-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd\" (UID: \"09418682-d550-4d4c-bf98-0b2e6fad1d1c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd"
Nov 03 19:41:17.023466 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.023441    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09418682-d550-4d4c-bf98-0b2e6fad1d1c-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd\" (UID: \"09418682-d550-4d4c-bf98-0b2e6fad1d1c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd"
Nov 03 19:41:17.023567 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.023489    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4d42\" (UniqueName: \"kubernetes.io/projected/09418682-d550-4d4c-bf98-0b2e6fad1d1c-kube-api-access-k4d42\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd\" (UID: \"09418682-d550-4d4c-bf98-0b2e6fad1d1c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd"
Nov 03 19:41:17.023848 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.023828    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z"]
Nov 03 19:41:17.027159 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.027145    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z"
Nov 03 19:41:17.033782 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.033416    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-fgw5t\""
Nov 03 19:41:17.038583 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.038561    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z"]
Nov 03 19:41:17.124334 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.124303    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dcafbe95-e88e-41c3-b124-f266d6bc86fa-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z\" (UID: \"dcafbe95-e88e-41c3-b124-f266d6bc86fa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z"
Nov 03 19:41:17.124504 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.124349    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9jxg\" (UniqueName: \"kubernetes.io/projected/dcafbe95-e88e-41c3-b124-f266d6bc86fa-kube-api-access-g9jxg\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z\" (UID: \"dcafbe95-e88e-41c3-b124-f266d6bc86fa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z"
Nov 03 19:41:17.124504 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.124426    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09418682-d550-4d4c-bf98-0b2e6fad1d1c-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd\" (UID: \"09418682-d550-4d4c-bf98-0b2e6fad1d1c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd"
Nov 03 19:41:17.124504 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.124474    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09418682-d550-4d4c-bf98-0b2e6fad1d1c-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd\" (UID: \"09418682-d550-4d4c-bf98-0b2e6fad1d1c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd"
Nov 03 19:41:17.124621 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.124526    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4d42\" (UniqueName: \"kubernetes.io/projected/09418682-d550-4d4c-bf98-0b2e6fad1d1c-kube-api-access-k4d42\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd\" (UID: \"09418682-d550-4d4c-bf98-0b2e6fad1d1c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd"
Nov 03 19:41:17.124621 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.124560    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dcafbe95-e88e-41c3-b124-f266d6bc86fa-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z\" (UID: \"dcafbe95-e88e-41c3-b124-f266d6bc86fa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z"
Nov 03 19:41:17.124880 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.124860    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09418682-d550-4d4c-bf98-0b2e6fad1d1c-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd\" (UID: \"09418682-d550-4d4c-bf98-0b2e6fad1d1c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd"
Nov 03 19:41:17.124917 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.124876    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09418682-d550-4d4c-bf98-0b2e6fad1d1c-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd\" (UID: \"09418682-d550-4d4c-bf98-0b2e6fad1d1c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd"
Nov 03 19:41:17.137182 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.137154    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4d42\" (UniqueName: \"kubernetes.io/projected/09418682-d550-4d4c-bf98-0b2e6fad1d1c-kube-api-access-k4d42\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd\" (UID: \"09418682-d550-4d4c-bf98-0b2e6fad1d1c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd"
Nov 03 19:41:17.225380 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.225308    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dcafbe95-e88e-41c3-b124-f266d6bc86fa-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z\" (UID: \"dcafbe95-e88e-41c3-b124-f266d6bc86fa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z"
Nov 03 19:41:17.225380 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.225364    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dcafbe95-e88e-41c3-b124-f266d6bc86fa-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z\" (UID: \"dcafbe95-e88e-41c3-b124-f266d6bc86fa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z"
Nov 03 19:41:17.225528 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.225404    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9jxg\" (UniqueName: \"kubernetes.io/projected/dcafbe95-e88e-41c3-b124-f266d6bc86fa-kube-api-access-g9jxg\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z\" (UID: \"dcafbe95-e88e-41c3-b124-f266d6bc86fa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z"
Nov 03 19:41:17.225715 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.225694    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dcafbe95-e88e-41c3-b124-f266d6bc86fa-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z\" (UID: \"dcafbe95-e88e-41c3-b124-f266d6bc86fa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z"
Nov 03 19:41:17.225755 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.225710    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dcafbe95-e88e-41c3-b124-f266d6bc86fa-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z\" (UID: \"dcafbe95-e88e-41c3-b124-f266d6bc86fa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z"
Nov 03 19:41:17.237539 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.237513    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9jxg\" (UniqueName: \"kubernetes.io/projected/dcafbe95-e88e-41c3-b124-f266d6bc86fa-kube-api-access-g9jxg\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z\" (UID: \"dcafbe95-e88e-41c3-b124-f266d6bc86fa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z"
Nov 03 19:41:17.244304 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.244285    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd"
Nov 03 19:41:17.342503 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.342476    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z"
Nov 03 19:41:17.374698 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.374668    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"]
Nov 03 19:41:17.378015 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.377991    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"
Nov 03 19:41:17.382402 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.382378    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 19:41:17.382529 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.382427    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 19:41:17.382529 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.382429    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 19:41:17.382529 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.382438    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-nh6bx\""
Nov 03 19:41:17.382529 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.382473    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 19:41:17.382529 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.382504    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 19:41:17.382680 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.382554    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 19:41:17.401247 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.396169    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"]
Nov 03 19:41:17.427160 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.427133    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-hw2zm\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"
Nov 03 19:41:17.427317 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.427171    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-hw2zm\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"
Nov 03 19:41:17.427317 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.427257    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-hw2zm\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"
Nov 03 19:41:17.427317 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.427281    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/72400fbd-e24b-4ece-a002-13a87c830ae0-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-hw2zm\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"
Nov 03 19:41:17.427317 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.427302    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/72400fbd-e24b-4ece-a002-13a87c830ae0-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-hw2zm\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"
Nov 03 19:41:17.427466 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.427395    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrw6w\" (UniqueName: \"kubernetes.io/projected/72400fbd-e24b-4ece-a002-13a87c830ae0-kube-api-access-nrw6w\") pod \"istiod-openshift-gateway-7876bfcb74-hw2zm\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"
Nov 03 19:41:17.427466 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.427445    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-hw2zm\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"
Nov 03 19:41:17.518977 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.518946    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd"]
Nov 03 19:41:17.521798 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:41:17.521776    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09418682_d550_4d4c_bf98_0b2e6fad1d1c.slice/crio-6f5c4912656f560c5be7f84cb74c42284923c1a1f7d4aa9d1aae0d582f8a1100 WatchSource:0}: Error finding container 6f5c4912656f560c5be7f84cb74c42284923c1a1f7d4aa9d1aae0d582f8a1100: Status 404 returned error can't find the container with id 6f5c4912656f560c5be7f84cb74c42284923c1a1f7d4aa9d1aae0d582f8a1100
Nov 03 19:41:17.523705 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.523690    5591 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:41:17.527974 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.527954    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/72400fbd-e24b-4ece-a002-13a87c830ae0-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-hw2zm\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"
Nov 03 19:41:17.528043 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.527986    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrw6w\" (UniqueName: \"kubernetes.io/projected/72400fbd-e24b-4ece-a002-13a87c830ae0-kube-api-access-nrw6w\") pod \"istiod-openshift-gateway-7876bfcb74-hw2zm\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"
Nov 03 19:41:17.528043 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.528008    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-hw2zm\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"
Nov 03 19:41:17.528107 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.528047    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-hw2zm\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"
Nov 03 19:41:17.528107 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.528080    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-hw2zm\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"
Nov 03 19:41:17.528185 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.528123    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-hw2zm\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"
Nov 03 19:41:17.528185 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.528153    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/72400fbd-e24b-4ece-a002-13a87c830ae0-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-hw2zm\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"
Nov 03 19:41:17.528789 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.528695    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-hw2zm\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"
Nov 03 19:41:17.530361 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.530340    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/72400fbd-e24b-4ece-a002-13a87c830ae0-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-hw2zm\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"
Nov 03 19:41:17.530538 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.530522    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-hw2zm\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"
Nov 03 19:41:17.530616 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.530594    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-hw2zm\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"
Nov 03 19:41:17.530862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.530848    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/72400fbd-e24b-4ece-a002-13a87c830ae0-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-hw2zm\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"
Nov 03 19:41:17.540070 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.540046    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrw6w\" (UniqueName: \"kubernetes.io/projected/72400fbd-e24b-4ece-a002-13a87c830ae0-kube-api-access-nrw6w\") pod \"istiod-openshift-gateway-7876bfcb74-hw2zm\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"
Nov 03 19:41:17.540144 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.540127    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-hw2zm\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"
Nov 03 19:41:17.616041 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.616021    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z"]
Nov 03 19:41:17.619090 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:41:17.619068    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcafbe95_e88e_41c3_b124_f266d6bc86fa.slice/crio-8691576dd520047cd9dfb749b259571efe7260182020938a67c98029f2871385 WatchSource:0}: Error finding container 8691576dd520047cd9dfb749b259571efe7260182020938a67c98029f2871385: Status 404 returned error can't find the container with id 8691576dd520047cd9dfb749b259571efe7260182020938a67c98029f2871385
Nov 03 19:41:17.697372 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.697343    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"
Nov 03 19:41:17.974619 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:17.974542    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"]
Nov 03 19:41:17.978726 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:41:17.978697    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72400fbd_e24b_4ece_a002_13a87c830ae0.slice/crio-fbeab33fc9bc3227c13560e19d70cd1f722346bc71e00671ad1c59b3654a2833 WatchSource:0}: Error finding container fbeab33fc9bc3227c13560e19d70cd1f722346bc71e00671ad1c59b3654a2833: Status 404 returned error can't find the container with id fbeab33fc9bc3227c13560e19d70cd1f722346bc71e00671ad1c59b3654a2833
Nov 03 19:41:18.185261 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:18.185222    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm" event={"ID":"72400fbd-e24b-4ece-a002-13a87c830ae0","Type":"ContainerStarted","Data":"fbeab33fc9bc3227c13560e19d70cd1f722346bc71e00671ad1c59b3654a2833"}
Nov 03 19:41:18.186841 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:18.186785    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z" event={"ID":"dcafbe95-e88e-41c3-b124-f266d6bc86fa","Type":"ContainerStarted","Data":"8691576dd520047cd9dfb749b259571efe7260182020938a67c98029f2871385"}
Nov 03 19:41:18.188213 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:18.188180    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd" event={"ID":"09418682-d550-4d4c-bf98-0b2e6fad1d1c","Type":"ContainerStarted","Data":"6f5c4912656f560c5be7f84cb74c42284923c1a1f7d4aa9d1aae0d582f8a1100"}
Nov 03 19:41:21.689348 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:21.689309    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:41:21.689785 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:21.689399    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:41:22.214543 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:22.214504    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm" event={"ID":"72400fbd-e24b-4ece-a002-13a87c830ae0","Type":"ContainerStarted","Data":"1364b86eb7b51fa33f42fb604bbad14fc2aabb5aa1a5ff78f4b9a709dd810b1c"}
Nov 03 19:41:22.214839 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:22.214774    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"
Nov 03 19:41:22.216493 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:22.216462    5591 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-hw2zm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:41:22.216611 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:22.216531    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm" podUID="72400fbd-e24b-4ece-a002-13a87c830ae0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:41:23.219885 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:23.219850    5591 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-hw2zm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:41:23.220429 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:23.219923    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm" podUID="72400fbd-e24b-4ece-a002-13a87c830ae0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:41:25.229575 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:25.229489    5591 generic.go:356] "Generic (PLEG): container finished" podID="dcafbe95-e88e-41c3-b124-f266d6bc86fa" containerID="f5b00959f7c6376751b8f83202e4f7e1dcd5db0bcd0a8fc86d67569fc0be8348" exitCode=0
Nov 03 19:41:25.229994 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:25.229579    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z" event={"ID":"dcafbe95-e88e-41c3-b124-f266d6bc86fa","Type":"ContainerDied","Data":"f5b00959f7c6376751b8f83202e4f7e1dcd5db0bcd0a8fc86d67569fc0be8348"}
Nov 03 19:41:25.231286 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:25.231221    5591 generic.go:356] "Generic (PLEG): container finished" podID="09418682-d550-4d4c-bf98-0b2e6fad1d1c" containerID="94a42e8fbfec26451f5f032b80734236c5f8acb9cfa5f2d8f3788eb46cf6bf36" exitCode=0
Nov 03 19:41:25.231286 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:25.231258    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd" event={"ID":"09418682-d550-4d4c-bf98-0b2e6fad1d1c","Type":"ContainerDied","Data":"94a42e8fbfec26451f5f032b80734236c5f8acb9cfa5f2d8f3788eb46cf6bf36"}
Nov 03 19:41:25.250461 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:25.250421    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm" podStartSLOduration=4.543067585 podStartE2EDuration="8.250406566s" totalImagesPullingTime="3.707338981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:22.240122214 +0000 UTC m=+231.012415232" watchObservedRunningTime="2026-11-03 19:41:25.250406566 +0000 UTC m=+234.022699584"
Nov 03 19:41:26.220417 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:26.220344    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"
Nov 03 19:41:26.236384 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:26.236352    5591 generic.go:356] "Generic (PLEG): container finished" podID="09418682-d550-4d4c-bf98-0b2e6fad1d1c" containerID="936c6218cf44fa3b8df488aa2b24b84092cf9216ff422f871d31f08020409dd6" exitCode=0
Nov 03 19:41:26.236731 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:26.236440    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd" event={"ID":"09418682-d550-4d4c-bf98-0b2e6fad1d1c","Type":"ContainerDied","Data":"936c6218cf44fa3b8df488aa2b24b84092cf9216ff422f871d31f08020409dd6"}
Nov 03 19:41:28.725991 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:41:28.725932    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:41:32.262668 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:32.262637    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd" event={"ID":"09418682-d550-4d4c-bf98-0b2e6fad1d1c","Type":"ContainerStarted","Data":"addf3e753df4a4a658c8247da87ffb7bf53d1a410505f5936327029a22328469"}
Nov 03 19:41:33.268526 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:33.268490    5591 generic.go:356] "Generic (PLEG): container finished" podID="09418682-d550-4d4c-bf98-0b2e6fad1d1c" containerID="addf3e753df4a4a658c8247da87ffb7bf53d1a410505f5936327029a22328469" exitCode=0
Nov 03 19:41:33.268992 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:33.268585    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd" event={"ID":"09418682-d550-4d4c-bf98-0b2e6fad1d1c","Type":"ContainerDied","Data":"addf3e753df4a4a658c8247da87ffb7bf53d1a410505f5936327029a22328469"}
Nov 03 19:41:34.397824 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:34.397803    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd"
Nov 03 19:41:34.482412 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:34.482375    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09418682-d550-4d4c-bf98-0b2e6fad1d1c-kube-api-access-k4d42\" (UniqueName: \"kubernetes.io/projected/09418682-d550-4d4c-bf98-0b2e6fad1d1c-kube-api-access-k4d42\") pod \"09418682-d550-4d4c-bf98-0b2e6fad1d1c\" (UID: \"09418682-d550-4d4c-bf98-0b2e6fad1d1c\") "
Nov 03 19:41:34.482557 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:34.482451    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09418682-d550-4d4c-bf98-0b2e6fad1d1c-util\" (UniqueName: \"kubernetes.io/empty-dir/09418682-d550-4d4c-bf98-0b2e6fad1d1c-util\") pod \"09418682-d550-4d4c-bf98-0b2e6fad1d1c\" (UID: \"09418682-d550-4d4c-bf98-0b2e6fad1d1c\") "
Nov 03 19:41:34.482621 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:34.482565    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09418682-d550-4d4c-bf98-0b2e6fad1d1c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/09418682-d550-4d4c-bf98-0b2e6fad1d1c-bundle\") pod \"09418682-d550-4d4c-bf98-0b2e6fad1d1c\" (UID: \"09418682-d550-4d4c-bf98-0b2e6fad1d1c\") "
Nov 03 19:41:34.483069 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:34.483044    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09418682-d550-4d4c-bf98-0b2e6fad1d1c-bundle" pod "09418682-d550-4d4c-bf98-0b2e6fad1d1c" (UID: "09418682-d550-4d4c-bf98-0b2e6fad1d1c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:41:34.485192 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:34.485173    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09418682-d550-4d4c-bf98-0b2e6fad1d1c-kube-api-access-k4d42" pod "09418682-d550-4d4c-bf98-0b2e6fad1d1c" (UID: "09418682-d550-4d4c-bf98-0b2e6fad1d1c"). InnerVolumeSpecName "kube-api-access-k4d42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:41:34.488082 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:34.488062    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09418682-d550-4d4c-bf98-0b2e6fad1d1c-util" pod "09418682-d550-4d4c-bf98-0b2e6fad1d1c" (UID: "09418682-d550-4d4c-bf98-0b2e6fad1d1c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:41:34.583631 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:34.583556    5591 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09418682-d550-4d4c-bf98-0b2e6fad1d1c-bundle\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:34.583631 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:34.583580    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4d42\" (UniqueName: \"kubernetes.io/projected/09418682-d550-4d4c-bf98-0b2e6fad1d1c-kube-api-access-k4d42\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:34.583631 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:34.583590    5591 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09418682-d550-4d4c-bf98-0b2e6fad1d1c-util\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:35.277089 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:35.277061    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd"
Nov 03 19:41:35.277273 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:35.277061    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vlbpd" event={"ID":"09418682-d550-4d4c-bf98-0b2e6fad1d1c","Type":"ContainerDied","Data":"6f5c4912656f560c5be7f84cb74c42284923c1a1f7d4aa9d1aae0d582f8a1100"}
Nov 03 19:41:35.277273 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:35.277183    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f5c4912656f560c5be7f84cb74c42284923c1a1f7d4aa9d1aae0d582f8a1100"
Nov 03 19:41:38.289362 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:38.289330    5591 generic.go:356] "Generic (PLEG): container finished" podID="dcafbe95-e88e-41c3-b124-f266d6bc86fa" containerID="0817907cdf8236abf6a98272446a113b5fa246249821567daccecf8332808906" exitCode=0
Nov 03 19:41:38.289843 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:38.289369    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z" event={"ID":"dcafbe95-e88e-41c3-b124-f266d6bc86fa","Type":"ContainerDied","Data":"0817907cdf8236abf6a98272446a113b5fa246249821567daccecf8332808906"}
Nov 03 19:41:39.294508 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:39.294475    5591 generic.go:356] "Generic (PLEG): container finished" podID="dcafbe95-e88e-41c3-b124-f266d6bc86fa" containerID="657e1446be6cfc24ad212f61b9d8140a8afa59047d66d20ba04b8e75296f4194" exitCode=0
Nov 03 19:41:39.294871 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:39.294562    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z" event={"ID":"dcafbe95-e88e-41c3-b124-f266d6bc86fa","Type":"ContainerDied","Data":"657e1446be6cfc24ad212f61b9d8140a8afa59047d66d20ba04b8e75296f4194"}
Nov 03 19:41:40.440914 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:40.440886    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z"
Nov 03 19:41:40.530063 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:40.530029    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcafbe95-e88e-41c3-b124-f266d6bc86fa-bundle\" (UniqueName: \"kubernetes.io/empty-dir/dcafbe95-e88e-41c3-b124-f266d6bc86fa-bundle\") pod \"dcafbe95-e88e-41c3-b124-f266d6bc86fa\" (UID: \"dcafbe95-e88e-41c3-b124-f266d6bc86fa\") "
Nov 03 19:41:40.530063 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:40.530065    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcafbe95-e88e-41c3-b124-f266d6bc86fa-util\" (UniqueName: \"kubernetes.io/empty-dir/dcafbe95-e88e-41c3-b124-f266d6bc86fa-util\") pod \"dcafbe95-e88e-41c3-b124-f266d6bc86fa\" (UID: \"dcafbe95-e88e-41c3-b124-f266d6bc86fa\") "
Nov 03 19:41:40.530267 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:40.530115    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcafbe95-e88e-41c3-b124-f266d6bc86fa-kube-api-access-g9jxg\" (UniqueName: \"kubernetes.io/projected/dcafbe95-e88e-41c3-b124-f266d6bc86fa-kube-api-access-g9jxg\") pod \"dcafbe95-e88e-41c3-b124-f266d6bc86fa\" (UID: \"dcafbe95-e88e-41c3-b124-f266d6bc86fa\") "
Nov 03 19:41:40.530761 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:40.530727    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcafbe95-e88e-41c3-b124-f266d6bc86fa-bundle" pod "dcafbe95-e88e-41c3-b124-f266d6bc86fa" (UID: "dcafbe95-e88e-41c3-b124-f266d6bc86fa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:41:40.532318 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:40.532297    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcafbe95-e88e-41c3-b124-f266d6bc86fa-kube-api-access-g9jxg" pod "dcafbe95-e88e-41c3-b124-f266d6bc86fa" (UID: "dcafbe95-e88e-41c3-b124-f266d6bc86fa"). InnerVolumeSpecName "kube-api-access-g9jxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:41:40.535124 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:40.535103    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcafbe95-e88e-41c3-b124-f266d6bc86fa-util" pod "dcafbe95-e88e-41c3-b124-f266d6bc86fa" (UID: "dcafbe95-e88e-41c3-b124-f266d6bc86fa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:41:40.631553 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:40.631472    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9jxg\" (UniqueName: \"kubernetes.io/projected/dcafbe95-e88e-41c3-b124-f266d6bc86fa-kube-api-access-g9jxg\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:40.631553 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:40.631503    5591 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dcafbe95-e88e-41c3-b124-f266d6bc86fa-bundle\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:40.631553 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:40.631512    5591 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dcafbe95-e88e-41c3-b124-f266d6bc86fa-util\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:41.303356 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:41.303321    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z" event={"ID":"dcafbe95-e88e-41c3-b124-f266d6bc86fa","Type":"ContainerDied","Data":"8691576dd520047cd9dfb749b259571efe7260182020938a67c98029f2871385"}
Nov 03 19:41:41.303356 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:41.303350    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ck7m4z"
Nov 03 19:41:41.303558 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:41:41.303377    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8691576dd520047cd9dfb749b259571efe7260182020938a67c98029f2871385"
Nov 03 19:41:58.722640 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:41:58.722598    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:42:00.674627 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.674586    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn"]
Nov 03 19:42:00.674998 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.674868    5591 cpu_manager.go:414] "containerMap: removing container" podUID="09418682-d550-4d4c-bf98-0b2e6fad1d1c" containerName="extract"
Nov 03 19:42:00.674998 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.674878    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09418682-d550-4d4c-bf98-0b2e6fad1d1c" containerName="extract"
Nov 03 19:42:00.674998 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.674891    5591 cpu_manager.go:414] "containerMap: removing container" podUID="dcafbe95-e88e-41c3-b124-f266d6bc86fa" containerName="extract"
Nov 03 19:42:00.674998 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.674896    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcafbe95-e88e-41c3-b124-f266d6bc86fa" containerName="extract"
Nov 03 19:42:00.674998 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.674909    5591 cpu_manager.go:414] "containerMap: removing container" podUID="dcafbe95-e88e-41c3-b124-f266d6bc86fa" containerName="pull"
Nov 03 19:42:00.674998 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.674914    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcafbe95-e88e-41c3-b124-f266d6bc86fa" containerName="pull"
Nov 03 19:42:00.674998 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.674921    5591 cpu_manager.go:414] "containerMap: removing container" podUID="09418682-d550-4d4c-bf98-0b2e6fad1d1c" containerName="pull"
Nov 03 19:42:00.674998 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.674926    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09418682-d550-4d4c-bf98-0b2e6fad1d1c" containerName="pull"
Nov 03 19:42:00.674998 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.674940    5591 cpu_manager.go:414] "containerMap: removing container" podUID="dcafbe95-e88e-41c3-b124-f266d6bc86fa" containerName="util"
Nov 03 19:42:00.674998 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.674944    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcafbe95-e88e-41c3-b124-f266d6bc86fa" containerName="util"
Nov 03 19:42:00.674998 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.674951    5591 cpu_manager.go:414] "containerMap: removing container" podUID="09418682-d550-4d4c-bf98-0b2e6fad1d1c" containerName="util"
Nov 03 19:42:00.674998 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.674956    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09418682-d550-4d4c-bf98-0b2e6fad1d1c" containerName="util"
Nov 03 19:42:00.674998 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.674998    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="09418682-d550-4d4c-bf98-0b2e6fad1d1c" containerName="extract"
Nov 03 19:42:00.675383 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.675006    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcafbe95-e88e-41c3-b124-f266d6bc86fa" containerName="extract"
Nov 03 19:42:00.677265 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.677250    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn"
Nov 03 19:42:00.719190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.719148    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn"]
Nov 03 19:42:00.756815 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.756775    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 19:42:00.757008 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.756829    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 19:42:00.757008 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.756864    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 19:42:00.757008 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.756887    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 19:42:00.757008 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.756910    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 19:42:00.757008 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.756990    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-847xh\""
Nov 03 19:42:00.796718 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.796689    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6f5l\" (UniqueName: \"kubernetes.io/projected/e0374e84-add2-4cdc-885a-4e1c130cedd7-kube-api-access-b6f5l\") pod \"opendatahub-operator-controller-manager-8464c976fd-8g6hn\" (UID: \"e0374e84-add2-4cdc-885a-4e1c130cedd7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn"
Nov 03 19:42:00.796879 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.796745    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e0374e84-add2-4cdc-885a-4e1c130cedd7-manifests\") pod \"opendatahub-operator-controller-manager-8464c976fd-8g6hn\" (UID: \"e0374e84-add2-4cdc-885a-4e1c130cedd7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn"
Nov 03 19:42:00.796879 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.796822    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0374e84-add2-4cdc-885a-4e1c130cedd7-metrics-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-8g6hn\" (UID: \"e0374e84-add2-4cdc-885a-4e1c130cedd7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn"
Nov 03 19:42:00.796879 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.796861    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0374e84-add2-4cdc-885a-4e1c130cedd7-tmp\") pod \"opendatahub-operator-controller-manager-8464c976fd-8g6hn\" (UID: \"e0374e84-add2-4cdc-885a-4e1c130cedd7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn"
Nov 03 19:42:00.796996 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.796885    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e0374e84-add2-4cdc-885a-4e1c130cedd7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-8g6hn\" (UID: \"e0374e84-add2-4cdc-885a-4e1c130cedd7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn"
Nov 03 19:42:00.796996 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.796903    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0374e84-add2-4cdc-885a-4e1c130cedd7-webhook-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-8g6hn\" (UID: \"e0374e84-add2-4cdc-885a-4e1c130cedd7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn"
Nov 03 19:42:00.898004 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.897966    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e0374e84-add2-4cdc-885a-4e1c130cedd7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-8g6hn\" (UID: \"e0374e84-add2-4cdc-885a-4e1c130cedd7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn"
Nov 03 19:42:00.898004 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.898005    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0374e84-add2-4cdc-885a-4e1c130cedd7-webhook-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-8g6hn\" (UID: \"e0374e84-add2-4cdc-885a-4e1c130cedd7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn"
Nov 03 19:42:00.898316 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.898026    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6f5l\" (UniqueName: \"kubernetes.io/projected/e0374e84-add2-4cdc-885a-4e1c130cedd7-kube-api-access-b6f5l\") pod \"opendatahub-operator-controller-manager-8464c976fd-8g6hn\" (UID: \"e0374e84-add2-4cdc-885a-4e1c130cedd7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn"
Nov 03 19:42:00.898316 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.898112    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e0374e84-add2-4cdc-885a-4e1c130cedd7-manifests\") pod \"opendatahub-operator-controller-manager-8464c976fd-8g6hn\" (UID: \"e0374e84-add2-4cdc-885a-4e1c130cedd7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn"
Nov 03 19:42:00.898316 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.898158    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0374e84-add2-4cdc-885a-4e1c130cedd7-metrics-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-8g6hn\" (UID: \"e0374e84-add2-4cdc-885a-4e1c130cedd7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn"
Nov 03 19:42:00.898316 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.898247    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0374e84-add2-4cdc-885a-4e1c130cedd7-tmp\") pod \"opendatahub-operator-controller-manager-8464c976fd-8g6hn\" (UID: \"e0374e84-add2-4cdc-885a-4e1c130cedd7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn"
Nov 03 19:42:00.898606 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.898576    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e0374e84-add2-4cdc-885a-4e1c130cedd7-manifests\") pod \"opendatahub-operator-controller-manager-8464c976fd-8g6hn\" (UID: \"e0374e84-add2-4cdc-885a-4e1c130cedd7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn"
Nov 03 19:42:00.900659 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.900630    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0374e84-add2-4cdc-885a-4e1c130cedd7-tmp\") pod \"opendatahub-operator-controller-manager-8464c976fd-8g6hn\" (UID: \"e0374e84-add2-4cdc-885a-4e1c130cedd7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn"
Nov 03 19:42:00.900780 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.900768    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e0374e84-add2-4cdc-885a-4e1c130cedd7-metrics-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-8g6hn\" (UID: \"e0374e84-add2-4cdc-885a-4e1c130cedd7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn"
Nov 03 19:42:00.900866 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.900848    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e0374e84-add2-4cdc-885a-4e1c130cedd7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-8g6hn\" (UID: \"e0374e84-add2-4cdc-885a-4e1c130cedd7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn"
Nov 03 19:42:00.900961 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.900940    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0374e84-add2-4cdc-885a-4e1c130cedd7-webhook-cert\") pod \"opendatahub-operator-controller-manager-8464c976fd-8g6hn\" (UID: \"e0374e84-add2-4cdc-885a-4e1c130cedd7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn"
Nov 03 19:42:00.911386 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.911364    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6f5l\" (UniqueName: \"kubernetes.io/projected/e0374e84-add2-4cdc-885a-4e1c130cedd7-kube-api-access-b6f5l\") pod \"opendatahub-operator-controller-manager-8464c976fd-8g6hn\" (UID: \"e0374e84-add2-4cdc-885a-4e1c130cedd7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn"
Nov 03 19:42:00.990869 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:00.990800    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn"
Nov 03 19:42:01.292976 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:01.292943    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn"]
Nov 03 19:42:01.297401 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:42:01.297380    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0374e84_add2_4cdc_885a_4e1c130cedd7.slice/crio-389299ad518bfb1a53a3462b209bb68a43cf719d753cda8d7daeb92262e2db6d WatchSource:0}: Error finding container 389299ad518bfb1a53a3462b209bb68a43cf719d753cda8d7daeb92262e2db6d: Status 404 returned error can't find the container with id 389299ad518bfb1a53a3462b209bb68a43cf719d753cda8d7daeb92262e2db6d
Nov 03 19:42:01.374007 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:01.373971    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn" event={"ID":"e0374e84-add2-4cdc-885a-4e1c130cedd7","Type":"ContainerStarted","Data":"389299ad518bfb1a53a3462b209bb68a43cf719d753cda8d7daeb92262e2db6d"}
Nov 03 19:42:05.389957 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:05.389922    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn" event={"ID":"e0374e84-add2-4cdc-885a-4e1c130cedd7","Type":"ContainerStarted","Data":"cbb1d0ac258fa2a971be096ab2f7db67fd440e54571472ca210c7983e489f77b"}
Nov 03 19:42:06.395358 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:06.395322    5591 generic.go:356] "Generic (PLEG): container finished" podID="e0374e84-add2-4cdc-885a-4e1c130cedd7" containerID="cbb1d0ac258fa2a971be096ab2f7db67fd440e54571472ca210c7983e489f77b" exitCode=0
Nov 03 19:42:06.395773 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:06.395417    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn" event={"ID":"e0374e84-add2-4cdc-885a-4e1c130cedd7","Type":"ContainerDied","Data":"cbb1d0ac258fa2a971be096ab2f7db67fd440e54571472ca210c7983e489f77b"}
Nov 03 19:42:07.401004 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:07.400965    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn" event={"ID":"e0374e84-add2-4cdc-885a-4e1c130cedd7","Type":"ContainerStarted","Data":"0f7ca02cdbfc8b79bfeee0713d0d363bcb805ea924ee9acbb8cee27cdcfca92d"}
Nov 03 19:42:07.401436 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:07.401080    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn"
Nov 03 19:42:07.453805 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:07.453745    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn" podStartSLOduration=2.645518873 podStartE2EDuration="7.453726841s" totalImagesPullingTime="4.398245035s" totalInitContainerRuntime="409.962933ms" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:07.451287726 +0000 UTC m=+276.223580744" watchObservedRunningTime="2026-11-03 19:42:07.453726841 +0000 UTC m=+276.226019861"
Nov 03 19:42:18.407496 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:18.407411    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8464c976fd-8g6hn"
Nov 03 19:42:26.983106 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:26.983067    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-k47sq"]
Nov 03 19:42:26.989499 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:26.989478    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k47sq"
Nov 03 19:42:27.008354 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:27.008326    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-k47sq"]
Nov 03 19:42:27.018488 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:27.018463    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 19:42:27.018664 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:27.018518    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 19:42:27.019367 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:27.019347    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-8q7lz\""
Nov 03 19:42:27.116018 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:27.115979    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl8k7\" (UniqueName: \"kubernetes.io/projected/6b473500-5e5e-4959-81bb-83b82dda443e-kube-api-access-xl8k7\") pod \"cert-manager-cainjector-5c57bbd4fc-k47sq\" (UID: \"6b473500-5e5e-4959-81bb-83b82dda443e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k47sq"
Nov 03 19:42:27.116018 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:27.116021    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b473500-5e5e-4959-81bb-83b82dda443e-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-k47sq\" (UID: \"6b473500-5e5e-4959-81bb-83b82dda443e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k47sq"
Nov 03 19:42:27.216846 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:27.216811    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b473500-5e5e-4959-81bb-83b82dda443e-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-k47sq\" (UID: \"6b473500-5e5e-4959-81bb-83b82dda443e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k47sq"
Nov 03 19:42:27.217009 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:27.216904    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl8k7\" (UniqueName: \"kubernetes.io/projected/6b473500-5e5e-4959-81bb-83b82dda443e-kube-api-access-xl8k7\") pod \"cert-manager-cainjector-5c57bbd4fc-k47sq\" (UID: \"6b473500-5e5e-4959-81bb-83b82dda443e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k47sq"
Nov 03 19:42:27.240093 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:27.240014    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b473500-5e5e-4959-81bb-83b82dda443e-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-k47sq\" (UID: \"6b473500-5e5e-4959-81bb-83b82dda443e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k47sq"
Nov 03 19:42:27.240093 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:27.240042    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl8k7\" (UniqueName: \"kubernetes.io/projected/6b473500-5e5e-4959-81bb-83b82dda443e-kube-api-access-xl8k7\") pod \"cert-manager-cainjector-5c57bbd4fc-k47sq\" (UID: \"6b473500-5e5e-4959-81bb-83b82dda443e\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k47sq"
Nov 03 19:42:27.318949 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:27.318914    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k47sq"
Nov 03 19:42:27.606807 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:27.606774    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-k47sq"]
Nov 03 19:42:27.609958 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:42:27.609925    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b473500_5e5e_4959_81bb_83b82dda443e.slice/crio-03a4c1f313edc8df2a9c35efebc3d0b13637c8654d0985df184d4afe7e1bed68 WatchSource:0}: Error finding container 03a4c1f313edc8df2a9c35efebc3d0b13637c8654d0985df184d4afe7e1bed68: Status 404 returned error can't find the container with id 03a4c1f313edc8df2a9c35efebc3d0b13637c8654d0985df184d4afe7e1bed68
Nov 03 19:42:28.477354 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:28.477309    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k47sq" event={"ID":"6b473500-5e5e-4959-81bb-83b82dda443e","Type":"ContainerStarted","Data":"03a4c1f313edc8df2a9c35efebc3d0b13637c8654d0985df184d4afe7e1bed68"}
Nov 03 19:42:28.726413 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:42:28.726373    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:42:31.382840 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:31.382814    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 19:42:31.383797 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:31.383781    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 19:42:31.491615 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:31.491581    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k47sq" event={"ID":"6b473500-5e5e-4959-81bb-83b82dda443e","Type":"ContainerStarted","Data":"aea33ab1bf9d3fe79248d408260e5862407012dbe981739adf42718df42967e6"}
Nov 03 19:42:31.551744 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:31.551500    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-k47sq" podStartSLOduration=2.682456101 podStartE2EDuration="5.551484097s" totalImagesPullingTime="2.869027996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:31.550964736 +0000 UTC m=+300.323257756" watchObservedRunningTime="2026-11-03 19:42:31.551484097 +0000 UTC m=+300.323777116"
Nov 03 19:42:37.898021 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:37.897992    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-79ddfb4884-rnf5q"]
Nov 03 19:42:37.902598 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:37.902576    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79ddfb4884-rnf5q"
Nov 03 19:42:37.916725 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:37.916705    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 19:42:37.924629 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:37.924609    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 19:42:37.925940 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:37.925918    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fkx6r\""
Nov 03 19:42:37.927046 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:37.927015    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79ddfb4884-rnf5q"]
Nov 03 19:42:38.017081 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:38.017046    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4cff30c-a64b-41d5-a6e8-62241d1c7a67-tls-certs\") pod \"kube-auth-proxy-79ddfb4884-rnf5q\" (UID: \"b4cff30c-a64b-41d5-a6e8-62241d1c7a67\") " pod="openshift-ingress/kube-auth-proxy-79ddfb4884-rnf5q"
Nov 03 19:42:38.017272 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:38.017095    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4cff30c-a64b-41d5-a6e8-62241d1c7a67-tmp\") pod \"kube-auth-proxy-79ddfb4884-rnf5q\" (UID: \"b4cff30c-a64b-41d5-a6e8-62241d1c7a67\") " pod="openshift-ingress/kube-auth-proxy-79ddfb4884-rnf5q"
Nov 03 19:42:38.017272 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:38.017252    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4b4c\" (UniqueName: \"kubernetes.io/projected/b4cff30c-a64b-41d5-a6e8-62241d1c7a67-kube-api-access-h4b4c\") pod \"kube-auth-proxy-79ddfb4884-rnf5q\" (UID: \"b4cff30c-a64b-41d5-a6e8-62241d1c7a67\") " pod="openshift-ingress/kube-auth-proxy-79ddfb4884-rnf5q"
Nov 03 19:42:38.118161 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:38.118126    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4cff30c-a64b-41d5-a6e8-62241d1c7a67-tls-certs\") pod \"kube-auth-proxy-79ddfb4884-rnf5q\" (UID: \"b4cff30c-a64b-41d5-a6e8-62241d1c7a67\") " pod="openshift-ingress/kube-auth-proxy-79ddfb4884-rnf5q"
Nov 03 19:42:38.118358 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:38.118177    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4cff30c-a64b-41d5-a6e8-62241d1c7a67-tmp\") pod \"kube-auth-proxy-79ddfb4884-rnf5q\" (UID: \"b4cff30c-a64b-41d5-a6e8-62241d1c7a67\") " pod="openshift-ingress/kube-auth-proxy-79ddfb4884-rnf5q"
Nov 03 19:42:38.118358 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:38.118250    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4b4c\" (UniqueName: \"kubernetes.io/projected/b4cff30c-a64b-41d5-a6e8-62241d1c7a67-kube-api-access-h4b4c\") pod \"kube-auth-proxy-79ddfb4884-rnf5q\" (UID: \"b4cff30c-a64b-41d5-a6e8-62241d1c7a67\") " pod="openshift-ingress/kube-auth-proxy-79ddfb4884-rnf5q"
Nov 03 19:42:38.120716 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:38.120692    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4cff30c-a64b-41d5-a6e8-62241d1c7a67-tmp\") pod \"kube-auth-proxy-79ddfb4884-rnf5q\" (UID: \"b4cff30c-a64b-41d5-a6e8-62241d1c7a67\") " pod="openshift-ingress/kube-auth-proxy-79ddfb4884-rnf5q"
Nov 03 19:42:38.121020 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:38.121004    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4cff30c-a64b-41d5-a6e8-62241d1c7a67-tls-certs\") pod \"kube-auth-proxy-79ddfb4884-rnf5q\" (UID: \"b4cff30c-a64b-41d5-a6e8-62241d1c7a67\") " pod="openshift-ingress/kube-auth-proxy-79ddfb4884-rnf5q"
Nov 03 19:42:38.141007 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:38.140979    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4b4c\" (UniqueName: \"kubernetes.io/projected/b4cff30c-a64b-41d5-a6e8-62241d1c7a67-kube-api-access-h4b4c\") pod \"kube-auth-proxy-79ddfb4884-rnf5q\" (UID: \"b4cff30c-a64b-41d5-a6e8-62241d1c7a67\") " pod="openshift-ingress/kube-auth-proxy-79ddfb4884-rnf5q"
Nov 03 19:42:38.213706 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:38.213632    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79ddfb4884-rnf5q"
Nov 03 19:42:38.504863 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:38.504763    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79ddfb4884-rnf5q"]
Nov 03 19:42:38.509243 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:42:38.509216    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4cff30c_a64b_41d5_a6e8_62241d1c7a67.slice/crio-ba10d32ee557535e4f71e7b86b3ba196a44627139a300ced55b6a9541f75420b WatchSource:0}: Error finding container ba10d32ee557535e4f71e7b86b3ba196a44627139a300ced55b6a9541f75420b: Status 404 returned error can't find the container with id ba10d32ee557535e4f71e7b86b3ba196a44627139a300ced55b6a9541f75420b
Nov 03 19:42:38.517239 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:38.517196    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79ddfb4884-rnf5q" event={"ID":"b4cff30c-a64b-41d5-a6e8-62241d1c7a67","Type":"ContainerStarted","Data":"ba10d32ee557535e4f71e7b86b3ba196a44627139a300ced55b6a9541f75420b"}
Nov 03 19:42:39.190141 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:39.190106    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:42:39.201848 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:39.201824    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"
Nov 03 19:42:39.210250 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:39.210199    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:42:39.220435 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:39.220409    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 19:42:39.220435 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:39.220434    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 19:42:39.220652 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:39.220476    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 19:42:39.233810 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:39.233778    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:42:39.295549 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:39.295519    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-579hd\""
Nov 03 19:42:39.328890 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:39.328852    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6grp\" (UniqueName: \"kubernetes.io/projected/ea095e27-20df-40bb-a4c4-3997bd5e9cff-kube-api-access-h6grp\") pod \"limitador-limitador-7c9b87d574-fz8hm\" (UID: \"ea095e27-20df-40bb-a4c4-3997bd5e9cff\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"
Nov 03 19:42:39.329068 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:39.328901    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ea095e27-20df-40bb-a4c4-3997bd5e9cff-config-file\") pod \"limitador-limitador-7c9b87d574-fz8hm\" (UID: \"ea095e27-20df-40bb-a4c4-3997bd5e9cff\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"
Nov 03 19:42:39.430299 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:39.430261    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6grp\" (UniqueName: \"kubernetes.io/projected/ea095e27-20df-40bb-a4c4-3997bd5e9cff-kube-api-access-h6grp\") pod \"limitador-limitador-7c9b87d574-fz8hm\" (UID: \"ea095e27-20df-40bb-a4c4-3997bd5e9cff\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"
Nov 03 19:42:39.430299 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:39.430305    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ea095e27-20df-40bb-a4c4-3997bd5e9cff-config-file\") pod \"limitador-limitador-7c9b87d574-fz8hm\" (UID: \"ea095e27-20df-40bb-a4c4-3997bd5e9cff\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"
Nov 03 19:42:39.430991 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:39.430967    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ea095e27-20df-40bb-a4c4-3997bd5e9cff-config-file\") pod \"limitador-limitador-7c9b87d574-fz8hm\" (UID: \"ea095e27-20df-40bb-a4c4-3997bd5e9cff\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"
Nov 03 19:42:39.452073 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:39.452004    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6grp\" (UniqueName: \"kubernetes.io/projected/ea095e27-20df-40bb-a4c4-3997bd5e9cff-kube-api-access-h6grp\") pod \"limitador-limitador-7c9b87d574-fz8hm\" (UID: \"ea095e27-20df-40bb-a4c4-3997bd5e9cff\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"
Nov 03 19:42:39.530817 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:39.530782    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"
Nov 03 19:42:39.836049 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:39.836018    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:42:39.839924 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:42:39.839894    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea095e27_20df_40bb_a4c4_3997bd5e9cff.slice/crio-e709a06707c23741947033b7629b1f9e6922d5314e4d0f66d4ff70743322afe0 WatchSource:0}: Error finding container e709a06707c23741947033b7629b1f9e6922d5314e4d0f66d4ff70743322afe0: Status 404 returned error can't find the container with id e709a06707c23741947033b7629b1f9e6922d5314e4d0f66d4ff70743322afe0
Nov 03 19:42:40.528601 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:40.528563    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-fz8hm" event={"ID":"ea095e27-20df-40bb-a4c4-3997bd5e9cff","Type":"ContainerStarted","Data":"e709a06707c23741947033b7629b1f9e6922d5314e4d0f66d4ff70743322afe0"}
Nov 03 19:42:41.534987 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:41.534949    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79ddfb4884-rnf5q" event={"ID":"b4cff30c-a64b-41d5-a6e8-62241d1c7a67","Type":"ContainerStarted","Data":"0b94d0f9d4d9068146dd600bf95f15d3ebbabebe7424e3de3c67d58d99488cb4"}
Nov 03 19:42:41.556025 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:41.555971    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-79ddfb4884-rnf5q" podStartSLOduration=1.94154881 podStartE2EDuration="4.555935663s" totalImagesPullingTime="2.614386853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:41.554609677 +0000 UTC m=+310.326902696" watchObservedRunningTime="2026-11-03 19:42:41.555935663 +0000 UTC m=+310.328228674"
Nov 03 19:42:43.545627 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:43.545588    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-fz8hm" event={"ID":"ea095e27-20df-40bb-a4c4-3997bd5e9cff","Type":"ContainerStarted","Data":"6118a493d933d94a6ed6608e88ff213417fd5bae3ffdd54491d6f74772745340"}
Nov 03 19:42:43.546022 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:43.545690    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"
Nov 03 19:42:43.565870 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:43.565813    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-fz8hm" podStartSLOduration=1.687347669 podStartE2EDuration="4.565797364s" totalImagesPullingTime="2.878449695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:43.564335323 +0000 UTC m=+312.336628338" watchObservedRunningTime="2026-11-03 19:42:43.565797364 +0000 UTC m=+312.338090765"
Nov 03 19:42:54.550804 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:42:54.550774    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"
Nov 03 19:42:58.722922 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:42:58.722877    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:43:21.006806 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.006749    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55f7f5fdbf-r82xn"]
Nov 03 19:43:21.014525 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.014499    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55f7f5fdbf-r82xn"
Nov 03 19:43:21.027443 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.027418    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 19:43:21.027443 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.027418    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 19:43:21.027932 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.027537    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xc6tf\""
Nov 03 19:43:21.029036 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.029000    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55f7f5fdbf-r82xn"]
Nov 03 19:43:21.029522 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.029504    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 19:43:21.029674 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.029585    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 19:43:21.029731 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.029588    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 19:43:21.073116 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.073075    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6vls\" (UniqueName: \"kubernetes.io/projected/7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d-kube-api-access-d6vls\") pod \"maas-controller-55f7f5fdbf-r82xn\" (UID: \"7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d\") " pod="opendatahub/maas-controller-55f7f5fdbf-r82xn"
Nov 03 19:43:21.073116 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.073117    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d-metrics-certs\") pod \"maas-controller-55f7f5fdbf-r82xn\" (UID: \"7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d\") " pod="opendatahub/maas-controller-55f7f5fdbf-r82xn"
Nov 03 19:43:21.073377 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.073250    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d-webhook-cert\") pod \"maas-controller-55f7f5fdbf-r82xn\" (UID: \"7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d\") " pod="opendatahub/maas-controller-55f7f5fdbf-r82xn"
Nov 03 19:43:21.173817 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.173780    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6vls\" (UniqueName: \"kubernetes.io/projected/7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d-kube-api-access-d6vls\") pod \"maas-controller-55f7f5fdbf-r82xn\" (UID: \"7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d\") " pod="opendatahub/maas-controller-55f7f5fdbf-r82xn"
Nov 03 19:43:21.173817 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.173818    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d-metrics-certs\") pod \"maas-controller-55f7f5fdbf-r82xn\" (UID: \"7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d\") " pod="opendatahub/maas-controller-55f7f5fdbf-r82xn"
Nov 03 19:43:21.174047 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.173952    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d-webhook-cert\") pod \"maas-controller-55f7f5fdbf-r82xn\" (UID: \"7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d\") " pod="opendatahub/maas-controller-55f7f5fdbf-r82xn"
Nov 03 19:43:21.176507 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.176478    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d-webhook-cert\") pod \"maas-controller-55f7f5fdbf-r82xn\" (UID: \"7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d\") " pod="opendatahub/maas-controller-55f7f5fdbf-r82xn"
Nov 03 19:43:21.176633 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.176589    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d-metrics-certs\") pod \"maas-controller-55f7f5fdbf-r82xn\" (UID: \"7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d\") " pod="opendatahub/maas-controller-55f7f5fdbf-r82xn"
Nov 03 19:43:21.186865 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.186833    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6vls\" (UniqueName: \"kubernetes.io/projected/7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d-kube-api-access-d6vls\") pod \"maas-controller-55f7f5fdbf-r82xn\" (UID: \"7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d\") " pod="opendatahub/maas-controller-55f7f5fdbf-r82xn"
Nov 03 19:43:21.327999 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.327958    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55f7f5fdbf-r82xn"
Nov 03 19:43:21.521318 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.521282    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55c765799f-2bvrd"]
Nov 03 19:43:21.525018 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.524995    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55c765799f-2bvrd"
Nov 03 19:43:21.545225 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.545177    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55c765799f-2bvrd"]
Nov 03 19:43:21.578627 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.578538    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scmm2\" (UniqueName: \"kubernetes.io/projected/7ea4d469-fff5-4de8-825a-7a76de2c46b2-kube-api-access-scmm2\") pod \"maas-controller-55c765799f-2bvrd\" (UID: \"7ea4d469-fff5-4de8-825a-7a76de2c46b2\") " pod="opendatahub/maas-controller-55c765799f-2bvrd"
Nov 03 19:43:21.578627 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.578591    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ea4d469-fff5-4de8-825a-7a76de2c46b2-metrics-certs\") pod \"maas-controller-55c765799f-2bvrd\" (UID: \"7ea4d469-fff5-4de8-825a-7a76de2c46b2\") " pod="opendatahub/maas-controller-55c765799f-2bvrd"
Nov 03 19:43:21.578824 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.578710    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ea4d469-fff5-4de8-825a-7a76de2c46b2-webhook-cert\") pod \"maas-controller-55c765799f-2bvrd\" (UID: \"7ea4d469-fff5-4de8-825a-7a76de2c46b2\") " pod="opendatahub/maas-controller-55c765799f-2bvrd"
Nov 03 19:43:21.609449 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.609395    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55f7f5fdbf-r82xn"]
Nov 03 19:43:21.612463 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:43:21.612434    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f1a9a5c_727f_458d_8b5d_1a4ec9e0c37d.slice/crio-ebd6e1c8361e1be705902d7fcfff3c3ad817ef69a4ca53ebb59943b5203c6158 WatchSource:0}: Error finding container ebd6e1c8361e1be705902d7fcfff3c3ad817ef69a4ca53ebb59943b5203c6158: Status 404 returned error can't find the container with id ebd6e1c8361e1be705902d7fcfff3c3ad817ef69a4ca53ebb59943b5203c6158
Nov 03 19:43:21.680145 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.680101    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ea4d469-fff5-4de8-825a-7a76de2c46b2-webhook-cert\") pod \"maas-controller-55c765799f-2bvrd\" (UID: \"7ea4d469-fff5-4de8-825a-7a76de2c46b2\") " pod="opendatahub/maas-controller-55c765799f-2bvrd"
Nov 03 19:43:21.680356 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.680155    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scmm2\" (UniqueName: \"kubernetes.io/projected/7ea4d469-fff5-4de8-825a-7a76de2c46b2-kube-api-access-scmm2\") pod \"maas-controller-55c765799f-2bvrd\" (UID: \"7ea4d469-fff5-4de8-825a-7a76de2c46b2\") " pod="opendatahub/maas-controller-55c765799f-2bvrd"
Nov 03 19:43:21.680356 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.680196    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ea4d469-fff5-4de8-825a-7a76de2c46b2-metrics-certs\") pod \"maas-controller-55c765799f-2bvrd\" (UID: \"7ea4d469-fff5-4de8-825a-7a76de2c46b2\") " pod="opendatahub/maas-controller-55c765799f-2bvrd"
Nov 03 19:43:21.682877 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.682842    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ea4d469-fff5-4de8-825a-7a76de2c46b2-webhook-cert\") pod \"maas-controller-55c765799f-2bvrd\" (UID: \"7ea4d469-fff5-4de8-825a-7a76de2c46b2\") " pod="opendatahub/maas-controller-55c765799f-2bvrd"
Nov 03 19:43:21.683027 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.682907    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ea4d469-fff5-4de8-825a-7a76de2c46b2-metrics-certs\") pod \"maas-controller-55c765799f-2bvrd\" (UID: \"7ea4d469-fff5-4de8-825a-7a76de2c46b2\") " pod="opendatahub/maas-controller-55c765799f-2bvrd"
Nov 03 19:43:21.693435 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.693406    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scmm2\" (UniqueName: \"kubernetes.io/projected/7ea4d469-fff5-4de8-825a-7a76de2c46b2-kube-api-access-scmm2\") pod \"maas-controller-55c765799f-2bvrd\" (UID: \"7ea4d469-fff5-4de8-825a-7a76de2c46b2\") " pod="opendatahub/maas-controller-55c765799f-2bvrd"
Nov 03 19:43:21.696045 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.696018    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55f7f5fdbf-r82xn" event={"ID":"7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d","Type":"ContainerStarted","Data":"ebd6e1c8361e1be705902d7fcfff3c3ad817ef69a4ca53ebb59943b5203c6158"}
Nov 03 19:43:21.838120 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:21.838019    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55c765799f-2bvrd"
Nov 03 19:43:22.112713 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:22.112631    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55c765799f-2bvrd"]
Nov 03 19:43:22.115964 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:43:22.115937    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea4d469_fff5_4de8_825a_7a76de2c46b2.slice/crio-1c3ac8573fc4d0a7e1c26df437725db746fffad0e8174883360074ad16a7efaf WatchSource:0}: Error finding container 1c3ac8573fc4d0a7e1c26df437725db746fffad0e8174883360074ad16a7efaf: Status 404 returned error can't find the container with id 1c3ac8573fc4d0a7e1c26df437725db746fffad0e8174883360074ad16a7efaf
Nov 03 19:43:22.703309 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:22.703258    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55c765799f-2bvrd" event={"ID":"7ea4d469-fff5-4de8-825a-7a76de2c46b2","Type":"ContainerStarted","Data":"1c3ac8573fc4d0a7e1c26df437725db746fffad0e8174883360074ad16a7efaf"}
Nov 03 19:43:25.719772 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:25.719733    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55c765799f-2bvrd" event={"ID":"7ea4d469-fff5-4de8-825a-7a76de2c46b2","Type":"ContainerStarted","Data":"4c6776abaeae656e3ead38c258b97bf6f12c41e559bedbddabe94a7865181005"}
Nov 03 19:43:25.720224 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:25.719807    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55c765799f-2bvrd"
Nov 03 19:43:25.721374 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:25.721347    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55f7f5fdbf-r82xn" event={"ID":"7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d","Type":"ContainerStarted","Data":"e547ccefb3b95550bcd080ab75a2aeb1d0d2a096800ee935cd72189238184277"}
Nov 03 19:43:25.721500 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:25.721484    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55f7f5fdbf-r82xn"
Nov 03 19:43:25.768465 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:25.768397    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-55c765799f-2bvrd" podStartSLOduration=1.717764426 podStartE2EDuration="4.768383462s" totalImagesPullingTime="3.050619036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:25.766869514 +0000 UTC m=+354.539162532" watchObservedRunningTime="2026-11-03 19:43:25.768383462 +0000 UTC m=+354.540676479"
Nov 03 19:43:28.722305 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:43:28.722266    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:43:35.980060 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:35.979998    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-55f7f5fdbf-r82xn" podStartSLOduration=12.434140145 podStartE2EDuration="15.979983862s" totalImagesPullingTime="3.545843717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:25.801921152 +0000 UTC m=+354.574214182" watchObservedRunningTime="2026-11-03 19:43:35.979983862 +0000 UTC m=+364.752276893"
Nov 03 19:43:35.980706 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:35.980686    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-vx6gv"]
Nov 03 19:43:35.984273 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:35.984216    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-vx6gv"
Nov 03 19:43:35.999154 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:35.999125    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-vx6gv"]
Nov 03 19:43:36.002275 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.002250    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 03 19:43:36.002553 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.002535    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-d4cfd\""
Nov 03 19:43:36.059759 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.059725    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-zpphn"]
Nov 03 19:43:36.063144 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.063122    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-zpphn"
Nov 03 19:43:36.076110 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.076050    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 03 19:43:36.076307 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.076073    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 03 19:43:36.076307 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.076167    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-74fgb\""
Nov 03 19:43:36.083478 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.083450    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-zpphn"]
Nov 03 19:43:36.116157 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.116117    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvnvw\" (UniqueName: \"kubernetes.io/projected/d7b3e7a0-4984-4088-a297-be74c8796194-kube-api-access-tvnvw\") pod \"odh-model-controller-769875db56-zpphn\" (UID: \"d7b3e7a0-4984-4088-a297-be74c8796194\") " pod="opendatahub/odh-model-controller-769875db56-zpphn"
Nov 03 19:43:36.116373 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.116167    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/67d4147a-49d0-43ad-983d-cc033e4fc21a-cert\") pod \"kserve-controller-manager-776cc66d47-vx6gv\" (UID: \"67d4147a-49d0-43ad-983d-cc033e4fc21a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vx6gv"
Nov 03 19:43:36.116373 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.116243    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n89wj\" (UniqueName: \"kubernetes.io/projected/67d4147a-49d0-43ad-983d-cc033e4fc21a-kube-api-access-n89wj\") pod \"kserve-controller-manager-776cc66d47-vx6gv\" (UID: \"67d4147a-49d0-43ad-983d-cc033e4fc21a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vx6gv"
Nov 03 19:43:36.116373 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.116274    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7b3e7a0-4984-4088-a297-be74c8796194-cert\") pod \"odh-model-controller-769875db56-zpphn\" (UID: \"d7b3e7a0-4984-4088-a297-be74c8796194\") " pod="opendatahub/odh-model-controller-769875db56-zpphn"
Nov 03 19:43:36.116373 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.116290    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7b3e7a0-4984-4088-a297-be74c8796194-metrics-certs\") pod \"odh-model-controller-769875db56-zpphn\" (UID: \"d7b3e7a0-4984-4088-a297-be74c8796194\") " pod="opendatahub/odh-model-controller-769875db56-zpphn"
Nov 03 19:43:36.217080 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.217041    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/67d4147a-49d0-43ad-983d-cc033e4fc21a-cert\") pod \"kserve-controller-manager-776cc66d47-vx6gv\" (UID: \"67d4147a-49d0-43ad-983d-cc033e4fc21a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vx6gv"
Nov 03 19:43:36.217276 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.217093    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n89wj\" (UniqueName: \"kubernetes.io/projected/67d4147a-49d0-43ad-983d-cc033e4fc21a-kube-api-access-n89wj\") pod \"kserve-controller-manager-776cc66d47-vx6gv\" (UID: \"67d4147a-49d0-43ad-983d-cc033e4fc21a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vx6gv"
Nov 03 19:43:36.217276 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.217120    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7b3e7a0-4984-4088-a297-be74c8796194-cert\") pod \"odh-model-controller-769875db56-zpphn\" (UID: \"d7b3e7a0-4984-4088-a297-be74c8796194\") " pod="opendatahub/odh-model-controller-769875db56-zpphn"
Nov 03 19:43:36.217276 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.217142    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7b3e7a0-4984-4088-a297-be74c8796194-metrics-certs\") pod \"odh-model-controller-769875db56-zpphn\" (UID: \"d7b3e7a0-4984-4088-a297-be74c8796194\") " pod="opendatahub/odh-model-controller-769875db56-zpphn"
Nov 03 19:43:36.217276 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.217258    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvnvw\" (UniqueName: \"kubernetes.io/projected/d7b3e7a0-4984-4088-a297-be74c8796194-kube-api-access-tvnvw\") pod \"odh-model-controller-769875db56-zpphn\" (UID: \"d7b3e7a0-4984-4088-a297-be74c8796194\") " pod="opendatahub/odh-model-controller-769875db56-zpphn"
Nov 03 19:43:36.219760 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.219734    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/67d4147a-49d0-43ad-983d-cc033e4fc21a-cert\") pod \"kserve-controller-manager-776cc66d47-vx6gv\" (UID: \"67d4147a-49d0-43ad-983d-cc033e4fc21a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vx6gv"
Nov 03 19:43:36.219876 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.219813    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7b3e7a0-4984-4088-a297-be74c8796194-metrics-certs\") pod \"odh-model-controller-769875db56-zpphn\" (UID: \"d7b3e7a0-4984-4088-a297-be74c8796194\") " pod="opendatahub/odh-model-controller-769875db56-zpphn"
Nov 03 19:43:36.219876 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.219860    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7b3e7a0-4984-4088-a297-be74c8796194-cert\") pod \"odh-model-controller-769875db56-zpphn\" (UID: \"d7b3e7a0-4984-4088-a297-be74c8796194\") " pod="opendatahub/odh-model-controller-769875db56-zpphn"
Nov 03 19:43:36.229387 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.229359    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvnvw\" (UniqueName: \"kubernetes.io/projected/d7b3e7a0-4984-4088-a297-be74c8796194-kube-api-access-tvnvw\") pod \"odh-model-controller-769875db56-zpphn\" (UID: \"d7b3e7a0-4984-4088-a297-be74c8796194\") " pod="opendatahub/odh-model-controller-769875db56-zpphn"
Nov 03 19:43:36.229913 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.229890    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n89wj\" (UniqueName: \"kubernetes.io/projected/67d4147a-49d0-43ad-983d-cc033e4fc21a-kube-api-access-n89wj\") pod \"kserve-controller-manager-776cc66d47-vx6gv\" (UID: \"67d4147a-49d0-43ad-983d-cc033e4fc21a\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vx6gv"
Nov 03 19:43:36.297717 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.297687    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-vx6gv"
Nov 03 19:43:36.376820 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.376788    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-zpphn"
Nov 03 19:43:36.532907 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.532834    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"]
Nov 03 19:43:36.537567 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.537550    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"
Nov 03 19:43:36.563600 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.563521    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"]
Nov 03 19:43:36.601151 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.601130    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-vx6gv"]
Nov 03 19:43:36.603630 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:43:36.603606    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d4147a_49d0_43ad_983d_cc033e4fc21a.slice/crio-c0c858bd863d8f90e2402dba3cb691422cb68bd0c0236b1f05f0e45dd14cce3c WatchSource:0}: Error finding container c0c858bd863d8f90e2402dba3cb691422cb68bd0c0236b1f05f0e45dd14cce3c: Status 404 returned error can't find the container with id c0c858bd863d8f90e2402dba3cb691422cb68bd0c0236b1f05f0e45dd14cce3c
Nov 03 19:43:36.620882 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.620850    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr6ts\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"
Nov 03 19:43:36.621015 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.620890    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k46j4\" (UniqueName: \"kubernetes.io/projected/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-kube-api-access-k46j4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr6ts\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"
Nov 03 19:43:36.621015 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.620976    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr6ts\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"
Nov 03 19:43:36.621015 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.621012    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr6ts\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"
Nov 03 19:43:36.621113 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.621065    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr6ts\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"
Nov 03 19:43:36.621113 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.621097    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr6ts\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"
Nov 03 19:43:36.621192 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.621173    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr6ts\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"
Nov 03 19:43:36.662523 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.662494    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-zpphn"]
Nov 03 19:43:36.664778 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:43:36.664742    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7b3e7a0_4984_4088_a297_be74c8796194.slice/crio-9661c2c8a9492505a1035407c959ae5ce9a65fe46d7023f0868cf0e2d437aada WatchSource:0}: Error finding container 9661c2c8a9492505a1035407c959ae5ce9a65fe46d7023f0868cf0e2d437aada: Status 404 returned error can't find the container with id 9661c2c8a9492505a1035407c959ae5ce9a65fe46d7023f0868cf0e2d437aada
Nov 03 19:43:36.721660 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.721623    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr6ts\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"
Nov 03 19:43:36.721660 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.721655    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr6ts\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"
Nov 03 19:43:36.721921 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.721683    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr6ts\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"
Nov 03 19:43:36.721921 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.721807    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr6ts\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"
Nov 03 19:43:36.721921 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.721872    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr6ts\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"
Nov 03 19:43:36.722077 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.721968    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr6ts\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"
Nov 03 19:43:36.722077 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.722012    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k46j4\" (UniqueName: \"kubernetes.io/projected/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-kube-api-access-k46j4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr6ts\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"
Nov 03 19:43:36.722793 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.722766    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr6ts\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"
Nov 03 19:43:36.724148 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.724124    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr6ts\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"
Nov 03 19:43:36.724311 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.724293    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr6ts\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"
Nov 03 19:43:36.724360 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.724336    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr6ts\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"
Nov 03 19:43:36.724395 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.724354    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr6ts\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"
Nov 03 19:43:36.728779 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.728760    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55c765799f-2bvrd"
Nov 03 19:43:36.728900 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.728845    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55f7f5fdbf-r82xn"
Nov 03 19:43:36.738728 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.738707    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr6ts\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"
Nov 03 19:43:36.739169 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.739148    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k46j4\" (UniqueName: \"kubernetes.io/projected/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-kube-api-access-k46j4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xr6ts\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"
Nov 03 19:43:36.766796 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.766769    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-zpphn" event={"ID":"d7b3e7a0-4984-4088-a297-be74c8796194","Type":"ContainerStarted","Data":"9661c2c8a9492505a1035407c959ae5ce9a65fe46d7023f0868cf0e2d437aada"}
Nov 03 19:43:36.767976 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.767955    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-vx6gv" event={"ID":"67d4147a-49d0-43ad-983d-cc033e4fc21a","Type":"ContainerStarted","Data":"c0c858bd863d8f90e2402dba3cb691422cb68bd0c0236b1f05f0e45dd14cce3c"}
Nov 03 19:43:36.839094 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.836831    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55f7f5fdbf-r82xn"]
Nov 03 19:43:36.839094 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.837144    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-55f7f5fdbf-r82xn" podUID="7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d" containerName="manager" containerID="cri-o://e547ccefb3b95550bcd080ab75a2aeb1d0d2a096800ee935cd72189238184277" gracePeriod=30
Nov 03 19:43:36.849339 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:36.849309    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"
Nov 03 19:43:37.072137 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:37.072107    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55f7f5fdbf-r82xn"
Nov 03 19:43:37.126107 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:37.125678    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d-metrics-certs\") pod \"7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d\" (UID: \"7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d\") "
Nov 03 19:43:37.126107 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:37.125768    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d-webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d-webhook-cert\") pod \"7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d\" (UID: \"7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d\") "
Nov 03 19:43:37.126107 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:37.125852    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d-kube-api-access-d6vls\" (UniqueName: \"kubernetes.io/projected/7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d-kube-api-access-d6vls\") pod \"7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d\" (UID: \"7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d\") "
Nov 03 19:43:37.128752 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:37.128712    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d-webhook-cert" pod "7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d" (UID: "7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:43:37.128892 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:37.128861    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d-kube-api-access-d6vls" pod "7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d" (UID: "7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d"). InnerVolumeSpecName "kube-api-access-d6vls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:43:37.129187 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:37.129078    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d-metrics-certs" pod "7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d" (UID: "7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:43:37.144149 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:37.144120    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"]
Nov 03 19:43:37.150527 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:43:37.150501    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f55488b_ab5f_4da2_bbbe_ef8c0fa41882.slice/crio-caa1f821f2693c9210cbd02beeea9926a92e5007449999784894a36399862fed WatchSource:0}: Error finding container caa1f821f2693c9210cbd02beeea9926a92e5007449999784894a36399862fed: Status 404 returned error can't find the container with id caa1f821f2693c9210cbd02beeea9926a92e5007449999784894a36399862fed
Nov 03 19:43:37.154114 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:37.154077    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:43:37.154221 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:37.154162    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:43:37.227089 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:37.227058    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:37.227331 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:37.227095    5591 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d-webhook-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:37.227331 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:37.227111    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6vls\" (UniqueName: \"kubernetes.io/projected/7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d-kube-api-access-d6vls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:37.782688 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:37.782642    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts" event={"ID":"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882","Type":"ContainerStarted","Data":"38c819eb177ad2bdeaed4a0680eb37f29014c680eb5fd1602ee4660db56159ae"}
Nov 03 19:43:37.782852 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:37.782718    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts" event={"ID":"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882","Type":"ContainerStarted","Data":"caa1f821f2693c9210cbd02beeea9926a92e5007449999784894a36399862fed"}
Nov 03 19:43:37.783728 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:37.783701    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"
Nov 03 19:43:37.786554 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:37.785721    5591 generic.go:356] "Generic (PLEG): container finished" podID="7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d" containerID="e547ccefb3b95550bcd080ab75a2aeb1d0d2a096800ee935cd72189238184277" exitCode=0
Nov 03 19:43:37.786554 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:37.785851    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55f7f5fdbf-r82xn" event={"ID":"7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d","Type":"ContainerDied","Data":"e547ccefb3b95550bcd080ab75a2aeb1d0d2a096800ee935cd72189238184277"}
Nov 03 19:43:37.786554 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:37.785895    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55f7f5fdbf-r82xn" event={"ID":"7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d","Type":"ContainerDied","Data":"ebd6e1c8361e1be705902d7fcfff3c3ad817ef69a4ca53ebb59943b5203c6158"}
Nov 03 19:43:37.786554 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:37.785932    5591 scope.go:122] "RemoveContainer" containerID="e547ccefb3b95550bcd080ab75a2aeb1d0d2a096800ee935cd72189238184277"
Nov 03 19:43:37.786554 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:37.786081    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55f7f5fdbf-r82xn"
Nov 03 19:43:37.802911 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:37.802763    5591 scope.go:122] "RemoveContainer" containerID="e547ccefb3b95550bcd080ab75a2aeb1d0d2a096800ee935cd72189238184277"
Nov 03 19:43:37.803429 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:43:37.803318    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e547ccefb3b95550bcd080ab75a2aeb1d0d2a096800ee935cd72189238184277\": container with ID starting with e547ccefb3b95550bcd080ab75a2aeb1d0d2a096800ee935cd72189238184277 not found: ID does not exist" containerID="e547ccefb3b95550bcd080ab75a2aeb1d0d2a096800ee935cd72189238184277"
Nov 03 19:43:37.803429 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:37.803360    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e547ccefb3b95550bcd080ab75a2aeb1d0d2a096800ee935cd72189238184277"} err="failed to get container status \"e547ccefb3b95550bcd080ab75a2aeb1d0d2a096800ee935cd72189238184277\": rpc error: code = NotFound desc = could not find container \"e547ccefb3b95550bcd080ab75a2aeb1d0d2a096800ee935cd72189238184277\": container with ID starting with e547ccefb3b95550bcd080ab75a2aeb1d0d2a096800ee935cd72189238184277 not found: ID does not exist"
Nov 03 19:43:37.828596 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:37.828027    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts" podStartSLOduration=1.828007156 podStartE2EDuration="1.828007156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:37.826741234 +0000 UTC m=+366.599034253" watchObservedRunningTime="2026-11-03 19:43:37.828007156 +0000 UTC m=+366.600300175"
Nov 03 19:43:37.855939 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:37.855667    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55f7f5fdbf-r82xn"]
Nov 03 19:43:37.871123 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:37.871077    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-55f7f5fdbf-r82xn"]
Nov 03 19:43:38.795445 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:38.795419    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"
Nov 03 19:43:38.868042 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:38.868012    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"]
Nov 03 19:43:38.868314 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:38.868274    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm" podUID="72400fbd-e24b-4ece-a002-13a87c830ae0" containerName="discovery" containerID="cri-o://1364b86eb7b51fa33f42fb604bbad14fc2aabb5aa1a5ff78f4b9a709dd810b1c" gracePeriod=30
Nov 03 19:43:39.150865 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.150835    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"
Nov 03 19:43:39.256233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.255676    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72400fbd-e24b-4ece-a002-13a87c830ae0-cacerts\" (UniqueName: \"kubernetes.io/secret/72400fbd-e24b-4ece-a002-13a87c830ae0-cacerts\") pod \"72400fbd-e24b-4ece-a002-13a87c830ae0\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") "
Nov 03 19:43:39.256233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.255733    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-csr-dns-cert\") pod \"72400fbd-e24b-4ece-a002-13a87c830ae0\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") "
Nov 03 19:43:39.256233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.255784    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72400fbd-e24b-4ece-a002-13a87c830ae0-kube-api-access-nrw6w\" (UniqueName: \"kubernetes.io/projected/72400fbd-e24b-4ece-a002-13a87c830ae0-kube-api-access-nrw6w\") pod \"72400fbd-e24b-4ece-a002-13a87c830ae0\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") "
Nov 03 19:43:39.256233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.255827    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-token\" (UniqueName: \"kubernetes.io/projected/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-token\") pod \"72400fbd-e24b-4ece-a002-13a87c830ae0\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") "
Nov 03 19:43:39.256233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.255881    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72400fbd-e24b-4ece-a002-13a87c830ae0-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/72400fbd-e24b-4ece-a002-13a87c830ae0-local-certs\") pod \"72400fbd-e24b-4ece-a002-13a87c830ae0\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") "
Nov 03 19:43:39.256233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.255922    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-csr-ca-configmap\") pod \"72400fbd-e24b-4ece-a002-13a87c830ae0\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") "
Nov 03 19:43:39.256233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.255940    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-kubeconfig\") pod \"72400fbd-e24b-4ece-a002-13a87c830ae0\" (UID: \"72400fbd-e24b-4ece-a002-13a87c830ae0\") "
Nov 03 19:43:39.257733 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.257685    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-csr-ca-configmap" pod "72400fbd-e24b-4ece-a002-13a87c830ae0" (UID: "72400fbd-e24b-4ece-a002-13a87c830ae0"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:43:39.259171 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.259131    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72400fbd-e24b-4ece-a002-13a87c830ae0-cacerts" pod "72400fbd-e24b-4ece-a002-13a87c830ae0" (UID: "72400fbd-e24b-4ece-a002-13a87c830ae0"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:43:39.259864 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.259832    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72400fbd-e24b-4ece-a002-13a87c830ae0-kube-api-access-nrw6w" pod "72400fbd-e24b-4ece-a002-13a87c830ae0" (UID: "72400fbd-e24b-4ece-a002-13a87c830ae0"). InnerVolumeSpecName "kube-api-access-nrw6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:43:39.260068 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.260046    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-token" pod "72400fbd-e24b-4ece-a002-13a87c830ae0" (UID: "72400fbd-e24b-4ece-a002-13a87c830ae0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:43:39.261571 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.261549    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-kubeconfig" pod "72400fbd-e24b-4ece-a002-13a87c830ae0" (UID: "72400fbd-e24b-4ece-a002-13a87c830ae0"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:43:39.261915 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.261883    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72400fbd-e24b-4ece-a002-13a87c830ae0-local-certs" pod "72400fbd-e24b-4ece-a002-13a87c830ae0" (UID: "72400fbd-e24b-4ece-a002-13a87c830ae0"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:43:39.261998 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.261969    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-csr-dns-cert" pod "72400fbd-e24b-4ece-a002-13a87c830ae0" (UID: "72400fbd-e24b-4ece-a002-13a87c830ae0"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:43:39.348728 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.348647    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d" path="/var/lib/kubelet/pods/7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d/volumes"
Nov 03 19:43:39.357156 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.356965    5591 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/72400fbd-e24b-4ece-a002-13a87c830ae0-local-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:39.357156 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.356997    5591 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-csr-ca-configmap\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:39.357156 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.357008    5591 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-kubeconfig\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:39.357156 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.357016    5591 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/72400fbd-e24b-4ece-a002-13a87c830ae0-cacerts\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:39.357156 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.357029    5591 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-csr-dns-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:39.357156 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.357043    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrw6w\" (UniqueName: \"kubernetes.io/projected/72400fbd-e24b-4ece-a002-13a87c830ae0-kube-api-access-nrw6w\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:39.357156 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.357057    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72400fbd-e24b-4ece-a002-13a87c830ae0-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:39.799230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.799184    5591 generic.go:356] "Generic (PLEG): container finished" podID="72400fbd-e24b-4ece-a002-13a87c830ae0" containerID="1364b86eb7b51fa33f42fb604bbad14fc2aabb5aa1a5ff78f4b9a709dd810b1c" exitCode=0
Nov 03 19:43:39.799636 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.799238    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm" event={"ID":"72400fbd-e24b-4ece-a002-13a87c830ae0","Type":"ContainerDied","Data":"1364b86eb7b51fa33f42fb604bbad14fc2aabb5aa1a5ff78f4b9a709dd810b1c"}
Nov 03 19:43:39.799636 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.799266    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"
Nov 03 19:43:39.799636 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.799292    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm" event={"ID":"72400fbd-e24b-4ece-a002-13a87c830ae0","Type":"ContainerDied","Data":"fbeab33fc9bc3227c13560e19d70cd1f722346bc71e00671ad1c59b3654a2833"}
Nov 03 19:43:39.799636 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.799325    5591 scope.go:122] "RemoveContainer" containerID="1364b86eb7b51fa33f42fb604bbad14fc2aabb5aa1a5ff78f4b9a709dd810b1c"
Nov 03 19:43:39.802187 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.802159    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-vx6gv" event={"ID":"67d4147a-49d0-43ad-983d-cc033e4fc21a","Type":"ContainerStarted","Data":"36ab098b722711f5183166206a626de13ca76e778f0390a254b9cc652f564d4a"}
Nov 03 19:43:39.802317 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.802240    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-vx6gv"
Nov 03 19:43:39.808538 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.808518    5591 scope.go:122] "RemoveContainer" containerID="1364b86eb7b51fa33f42fb604bbad14fc2aabb5aa1a5ff78f4b9a709dd810b1c"
Nov 03 19:43:39.808857 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:43:39.808830    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1364b86eb7b51fa33f42fb604bbad14fc2aabb5aa1a5ff78f4b9a709dd810b1c\": container with ID starting with 1364b86eb7b51fa33f42fb604bbad14fc2aabb5aa1a5ff78f4b9a709dd810b1c not found: ID does not exist" containerID="1364b86eb7b51fa33f42fb604bbad14fc2aabb5aa1a5ff78f4b9a709dd810b1c"
Nov 03 19:43:39.808937 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.808863    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1364b86eb7b51fa33f42fb604bbad14fc2aabb5aa1a5ff78f4b9a709dd810b1c"} err="failed to get container status \"1364b86eb7b51fa33f42fb604bbad14fc2aabb5aa1a5ff78f4b9a709dd810b1c\": rpc error: code = NotFound desc = could not find container \"1364b86eb7b51fa33f42fb604bbad14fc2aabb5aa1a5ff78f4b9a709dd810b1c\": container with ID starting with 1364b86eb7b51fa33f42fb604bbad14fc2aabb5aa1a5ff78f4b9a709dd810b1c not found: ID does not exist"
Nov 03 19:43:39.825158 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.825107    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-vx6gv" podStartSLOduration=1.9739220130000001 podStartE2EDuration="4.825092531s" totalImagesPullingTime="2.851170518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:39.823865913 +0000 UTC m=+368.596158932" watchObservedRunningTime="2026-11-03 19:43:39.825092531 +0000 UTC m=+368.597385551"
Nov 03 19:43:39.839981 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.839959    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"]
Nov 03 19:43:39.844169 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:39.844147    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-hw2zm"]
Nov 03 19:43:40.811509 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:40.811473    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-zpphn" event={"ID":"d7b3e7a0-4984-4088-a297-be74c8796194","Type":"ContainerStarted","Data":"55649a7649ceb35243232d9fe172c1107b039989eab17949cacc7b9667d74125"}
Nov 03 19:43:40.811951 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:40.811656    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-zpphn"
Nov 03 19:43:40.852702 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:40.852638    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-zpphn" podStartSLOduration=1.7227242569999999 podStartE2EDuration="4.852624142s" totalImagesPullingTime="3.129899885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:40.852387953 +0000 UTC m=+369.624680971" watchObservedRunningTime="2026-11-03 19:43:40.852624142 +0000 UTC m=+369.624917189"
Nov 03 19:43:41.348099 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:41.348020    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72400fbd-e24b-4ece-a002-13a87c830ae0" path="/var/lib/kubelet/pods/72400fbd-e24b-4ece-a002-13a87c830ae0/volumes"
Nov 03 19:43:51.820028 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:51.819998    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-zpphn"
Nov 03 19:43:58.218219 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.218176    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl"]
Nov 03 19:43:58.218703 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.218684    5591 cpu_manager.go:414] "containerMap: removing container" podUID="7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d" containerName="manager"
Nov 03 19:43:58.218788 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.218702    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d" containerName="manager"
Nov 03 19:43:58.218788 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.218727    5591 cpu_manager.go:414] "containerMap: removing container" podUID="72400fbd-e24b-4ece-a002-13a87c830ae0" containerName="discovery"
Nov 03 19:43:58.218788 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.218737    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72400fbd-e24b-4ece-a002-13a87c830ae0" containerName="discovery"
Nov 03 19:43:58.218929 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.218855    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f1a9a5c-727f-458d-8b5d-1a4ec9e0c37d" containerName="manager"
Nov 03 19:43:58.218929 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.218870    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="72400fbd-e24b-4ece-a002-13a87c830ae0" containerName="discovery"
Nov 03 19:43:58.223963 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.223937    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl"
Nov 03 19:43:58.233137 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.233113    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 19:43:58.233939 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.233346    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:43:58.234608 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.234588    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-942q2\""
Nov 03 19:43:58.234697 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.234647    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:43:58.235775 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.235752    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 19:43:58.236269 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.236250    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl"]
Nov 03 19:43:58.319550 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.319511    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc55f85-5018-437a-8682-869fcbd05c58-openshift-service-ca\") pod \"maas-api-6f6fb65589-hczzl\" (UID: \"fdc55f85-5018-437a-8682-869fcbd05c58\") " pod="odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl"
Nov 03 19:43:58.319550 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.319549    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc55f85-5018-437a-8682-869fcbd05c58-metrics-certs\") pod \"maas-api-6f6fb65589-hczzl\" (UID: \"fdc55f85-5018-437a-8682-869fcbd05c58\") " pod="odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl"
Nov 03 19:43:58.319865 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.319625    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdc55f85-5018-437a-8682-869fcbd05c58-maas-api-tls\") pod \"maas-api-6f6fb65589-hczzl\" (UID: \"fdc55f85-5018-437a-8682-869fcbd05c58\") " pod="odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl"
Nov 03 19:43:58.319865 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.319730    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gntrw\" (UniqueName: \"kubernetes.io/projected/fdc55f85-5018-437a-8682-869fcbd05c58-kube-api-access-gntrw\") pod \"maas-api-6f6fb65589-hczzl\" (UID: \"fdc55f85-5018-437a-8682-869fcbd05c58\") " pod="odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl"
Nov 03 19:43:58.420301 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.420268    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdc55f85-5018-437a-8682-869fcbd05c58-maas-api-tls\") pod \"maas-api-6f6fb65589-hczzl\" (UID: \"fdc55f85-5018-437a-8682-869fcbd05c58\") " pod="odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl"
Nov 03 19:43:58.420465 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.420318    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gntrw\" (UniqueName: \"kubernetes.io/projected/fdc55f85-5018-437a-8682-869fcbd05c58-kube-api-access-gntrw\") pod \"maas-api-6f6fb65589-hczzl\" (UID: \"fdc55f85-5018-437a-8682-869fcbd05c58\") " pod="odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl"
Nov 03 19:43:58.420465 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.420368    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc55f85-5018-437a-8682-869fcbd05c58-openshift-service-ca\") pod \"maas-api-6f6fb65589-hczzl\" (UID: \"fdc55f85-5018-437a-8682-869fcbd05c58\") " pod="odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl"
Nov 03 19:43:58.420465 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.420385    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc55f85-5018-437a-8682-869fcbd05c58-metrics-certs\") pod \"maas-api-6f6fb65589-hczzl\" (UID: \"fdc55f85-5018-437a-8682-869fcbd05c58\") " pod="odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl"
Nov 03 19:43:58.421049 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.421016    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc55f85-5018-437a-8682-869fcbd05c58-openshift-service-ca\") pod \"maas-api-6f6fb65589-hczzl\" (UID: \"fdc55f85-5018-437a-8682-869fcbd05c58\") " pod="odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl"
Nov 03 19:43:58.423036 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.423015    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc55f85-5018-437a-8682-869fcbd05c58-metrics-certs\") pod \"maas-api-6f6fb65589-hczzl\" (UID: \"fdc55f85-5018-437a-8682-869fcbd05c58\") " pod="odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl"
Nov 03 19:43:58.423151 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.423058    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdc55f85-5018-437a-8682-869fcbd05c58-maas-api-tls\") pod \"maas-api-6f6fb65589-hczzl\" (UID: \"fdc55f85-5018-437a-8682-869fcbd05c58\") " pod="odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl"
Nov 03 19:43:58.435071 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.435042    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gntrw\" (UniqueName: \"kubernetes.io/projected/fdc55f85-5018-437a-8682-869fcbd05c58-kube-api-access-gntrw\") pod \"maas-api-6f6fb65589-hczzl\" (UID: \"fdc55f85-5018-437a-8682-869fcbd05c58\") " pod="odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl"
Nov 03 19:43:58.539172 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.539136    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl"
Nov 03 19:43:58.722841 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:43:58.722804    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:43:58.820648 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.820416    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl"]
Nov 03 19:43:58.823448 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:43:58.823418    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc55f85_5018_437a_8682_869fcbd05c58.slice/crio-39ba0e39d7b279232f867eb8e94d33193a9d813666e49d6ce03e06ac8b20d82f WatchSource:0}: Error finding container 39ba0e39d7b279232f867eb8e94d33193a9d813666e49d6ce03e06ac8b20d82f: Status 404 returned error can't find the container with id 39ba0e39d7b279232f867eb8e94d33193a9d813666e49d6ce03e06ac8b20d82f
Nov 03 19:43:58.893154 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:58.893119    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl" event={"ID":"fdc55f85-5018-437a-8682-869fcbd05c58","Type":"ContainerStarted","Data":"39ba0e39d7b279232f867eb8e94d33193a9d813666e49d6ce03e06ac8b20d82f"}
Nov 03 19:43:59.141491 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:43:59.141406    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:44:03.916433 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:03.916398    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl" event={"ID":"fdc55f85-5018-437a-8682-869fcbd05c58","Type":"ContainerStarted","Data":"422e50982149e0c7ea9a2820ad6884e9a227755432c0c825cd9c9ecbb99678bf"}
Nov 03 19:44:03.916892 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:03.916463    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl"
Nov 03 19:44:03.937826 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:03.937773    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl" podStartSLOduration=1.71856017 podStartE2EDuration="5.937757861s" totalImagesPullingTime="4.219197691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:03.936426855 +0000 UTC m=+392.708719895" watchObservedRunningTime="2026-11-03 19:44:03.937757861 +0000 UTC m=+392.710050878"
Nov 03 19:44:09.925930 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:09.925903    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl"
Nov 03 19:44:10.817897 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:10.817868    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-vx6gv"
Nov 03 19:44:20.960641 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:20.960605    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl"]
Nov 03 19:44:20.961140 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:20.960839    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl" podUID="fdc55f85-5018-437a-8682-869fcbd05c58" containerName="maas-api" containerID="cri-o://422e50982149e0c7ea9a2820ad6884e9a227755432c0c825cd9c9ecbb99678bf" gracePeriod=30
Nov 03 19:44:21.198637 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:21.198611    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl"
Nov 03 19:44:21.285007 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:21.284973    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdc55f85-5018-437a-8682-869fcbd05c58-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc55f85-5018-437a-8682-869fcbd05c58-metrics-certs\") pod \"fdc55f85-5018-437a-8682-869fcbd05c58\" (UID: \"fdc55f85-5018-437a-8682-869fcbd05c58\") "
Nov 03 19:44:21.285198 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:21.285023    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdc55f85-5018-437a-8682-869fcbd05c58-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdc55f85-5018-437a-8682-869fcbd05c58-maas-api-tls\") pod \"fdc55f85-5018-437a-8682-869fcbd05c58\" (UID: \"fdc55f85-5018-437a-8682-869fcbd05c58\") "
Nov 03 19:44:21.285198 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:21.285068    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdc55f85-5018-437a-8682-869fcbd05c58-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc55f85-5018-437a-8682-869fcbd05c58-openshift-service-ca\") pod \"fdc55f85-5018-437a-8682-869fcbd05c58\" (UID: \"fdc55f85-5018-437a-8682-869fcbd05c58\") "
Nov 03 19:44:21.285198 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:21.285089    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdc55f85-5018-437a-8682-869fcbd05c58-kube-api-access-gntrw\" (UniqueName: \"kubernetes.io/projected/fdc55f85-5018-437a-8682-869fcbd05c58-kube-api-access-gntrw\") pod \"fdc55f85-5018-437a-8682-869fcbd05c58\" (UID: \"fdc55f85-5018-437a-8682-869fcbd05c58\") "
Nov 03 19:44:21.285722 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:21.285538    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc55f85-5018-437a-8682-869fcbd05c58-openshift-service-ca" pod "fdc55f85-5018-437a-8682-869fcbd05c58" (UID: "fdc55f85-5018-437a-8682-869fcbd05c58"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:44:21.287664 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:21.287623    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc55f85-5018-437a-8682-869fcbd05c58-kube-api-access-gntrw" pod "fdc55f85-5018-437a-8682-869fcbd05c58" (UID: "fdc55f85-5018-437a-8682-869fcbd05c58"). InnerVolumeSpecName "kube-api-access-gntrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:44:21.287768 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:21.287684    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdc55f85-5018-437a-8682-869fcbd05c58-metrics-certs" pod "fdc55f85-5018-437a-8682-869fcbd05c58" (UID: "fdc55f85-5018-437a-8682-869fcbd05c58"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:44:21.287768 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:21.287709    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdc55f85-5018-437a-8682-869fcbd05c58-maas-api-tls" pod "fdc55f85-5018-437a-8682-869fcbd05c58" (UID: "fdc55f85-5018-437a-8682-869fcbd05c58"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:44:21.385673 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:21.385641    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdc55f85-5018-437a-8682-869fcbd05c58-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:21.385673 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:21.385665    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdc55f85-5018-437a-8682-869fcbd05c58-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:21.385673 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:21.385676    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdc55f85-5018-437a-8682-869fcbd05c58-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:21.385893 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:21.385685    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gntrw\" (UniqueName: \"kubernetes.io/projected/fdc55f85-5018-437a-8682-869fcbd05c58-kube-api-access-gntrw\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:21.996998 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:21.996960    5591 generic.go:356] "Generic (PLEG): container finished" podID="fdc55f85-5018-437a-8682-869fcbd05c58" containerID="422e50982149e0c7ea9a2820ad6884e9a227755432c0c825cd9c9ecbb99678bf" exitCode=0
Nov 03 19:44:21.997445 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:21.997004    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl"
Nov 03 19:44:21.997445 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:21.997034    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl" event={"ID":"fdc55f85-5018-437a-8682-869fcbd05c58","Type":"ContainerDied","Data":"422e50982149e0c7ea9a2820ad6884e9a227755432c0c825cd9c9ecbb99678bf"}
Nov 03 19:44:21.997445 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:21.997095    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl" event={"ID":"fdc55f85-5018-437a-8682-869fcbd05c58","Type":"ContainerDied","Data":"39ba0e39d7b279232f867eb8e94d33193a9d813666e49d6ce03e06ac8b20d82f"}
Nov 03 19:44:21.997445 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:21.997124    5591 scope.go:122] "RemoveContainer" containerID="422e50982149e0c7ea9a2820ad6884e9a227755432c0c825cd9c9ecbb99678bf"
Nov 03 19:44:22.005788 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:22.005769    5591 scope.go:122] "RemoveContainer" containerID="422e50982149e0c7ea9a2820ad6884e9a227755432c0c825cd9c9ecbb99678bf"
Nov 03 19:44:22.006102 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:44:22.006078    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"422e50982149e0c7ea9a2820ad6884e9a227755432c0c825cd9c9ecbb99678bf\": container with ID starting with 422e50982149e0c7ea9a2820ad6884e9a227755432c0c825cd9c9ecbb99678bf not found: ID does not exist" containerID="422e50982149e0c7ea9a2820ad6884e9a227755432c0c825cd9c9ecbb99678bf"
Nov 03 19:44:22.006181 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:22.006109    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"422e50982149e0c7ea9a2820ad6884e9a227755432c0c825cd9c9ecbb99678bf"} err="failed to get container status \"422e50982149e0c7ea9a2820ad6884e9a227755432c0c825cd9c9ecbb99678bf\": rpc error: code = NotFound desc = could not find container \"422e50982149e0c7ea9a2820ad6884e9a227755432c0c825cd9c9ecbb99678bf\": container with ID starting with 422e50982149e0c7ea9a2820ad6884e9a227755432c0c825cd9c9ecbb99678bf not found: ID does not exist"
Nov 03 19:44:22.014344 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:22.014322    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl"]
Nov 03 19:44:22.021523 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:22.021504    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f6fb65589-hczzl"]
Nov 03 19:44:23.346676 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:44:23.346644    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdc55f85-5018-437a-8682-869fcbd05c58" path="/var/lib/kubelet/pods/fdc55f85-5018-437a-8682-869fcbd05c58/volumes"
Nov 03 19:44:28.722830 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:44:28.722795    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:44:58.721955 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:44:58.721915    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:45:00.143869 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:00.143835    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-wsk95"]
Nov 03 19:45:00.144338 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:00.144176    5591 cpu_manager.go:414] "containerMap: removing container" podUID="fdc55f85-5018-437a-8682-869fcbd05c58" containerName="maas-api"
Nov 03 19:45:00.144338 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:00.144187    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdc55f85-5018-437a-8682-869fcbd05c58" containerName="maas-api"
Nov 03 19:45:00.144338 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:00.144287    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdc55f85-5018-437a-8682-869fcbd05c58" containerName="maas-api"
Nov 03 19:45:00.147193 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:00.147176    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-wsk95"
Nov 03 19:45:00.150959 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:00.150932    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:45:00.150959 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:00.150935    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:45:00.152035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:00.152022    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\""
Nov 03 19:45:00.155791 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:00.155767    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-wsk95"]
Nov 03 19:45:00.309000 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:00.308961    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd31e715-d9ba-4470-8550-3381c232070d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895585-wsk95\" (UID: \"fd31e715-d9ba-4470-8550-3381c232070d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-wsk95"
Nov 03 19:45:00.309189 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:00.309007    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbfrn\" (UniqueName: \"kubernetes.io/projected/fd31e715-d9ba-4470-8550-3381c232070d-kube-api-access-vbfrn\") pod \"maas-api-key-cleanup-29895585-wsk95\" (UID: \"fd31e715-d9ba-4470-8550-3381c232070d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-wsk95"
Nov 03 19:45:00.409585 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:00.409488    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd31e715-d9ba-4470-8550-3381c232070d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895585-wsk95\" (UID: \"fd31e715-d9ba-4470-8550-3381c232070d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-wsk95"
Nov 03 19:45:00.409585 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:00.409537    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbfrn\" (UniqueName: \"kubernetes.io/projected/fd31e715-d9ba-4470-8550-3381c232070d-kube-api-access-vbfrn\") pod \"maas-api-key-cleanup-29895585-wsk95\" (UID: \"fd31e715-d9ba-4470-8550-3381c232070d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-wsk95"
Nov 03 19:45:00.410264 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:00.410239    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd31e715-d9ba-4470-8550-3381c232070d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895585-wsk95\" (UID: \"fd31e715-d9ba-4470-8550-3381c232070d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-wsk95"
Nov 03 19:45:00.422718 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:00.422696    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbfrn\" (UniqueName: \"kubernetes.io/projected/fd31e715-d9ba-4470-8550-3381c232070d-kube-api-access-vbfrn\") pod \"maas-api-key-cleanup-29895585-wsk95\" (UID: \"fd31e715-d9ba-4470-8550-3381c232070d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-wsk95"
Nov 03 19:45:00.460617 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:00.460590    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-wsk95"
Nov 03 19:45:00.734903 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:00.734823    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-wsk95"]
Nov 03 19:45:00.737833 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:45:00.737807    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd31e715_d9ba_4470_8550_3381c232070d.slice/crio-c8e7a9107414098a037abbe05a73d70d3897e11201c34e67bf558be375a0573d WatchSource:0}: Error finding container c8e7a9107414098a037abbe05a73d70d3897e11201c34e67bf558be375a0573d: Status 404 returned error can't find the container with id c8e7a9107414098a037abbe05a73d70d3897e11201c34e67bf558be375a0573d
Nov 03 19:45:01.155079 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:01.155043    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-wsk95" event={"ID":"fd31e715-d9ba-4470-8550-3381c232070d","Type":"ContainerStarted","Data":"c8e7a9107414098a037abbe05a73d70d3897e11201c34e67bf558be375a0573d"}
Nov 03 19:45:12.206386 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:12.206343    5591 generic.go:356] "Generic (PLEG): container finished" podID="fd31e715-d9ba-4470-8550-3381c232070d" containerID="b79a48061b327e92a42f5cdbef14dafe3dd9f1f38cec0ba60ea63a236f46ab1b" exitCode=60
Nov 03 19:45:12.206795 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:12.206401    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-wsk95" event={"ID":"fd31e715-d9ba-4470-8550-3381c232070d","Type":"ContainerDied","Data":"b79a48061b327e92a42f5cdbef14dafe3dd9f1f38cec0ba60ea63a236f46ab1b"}
Nov 03 19:45:12.206795 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:12.206726    5591 scope.go:122] "RemoveContainer" containerID="b79a48061b327e92a42f5cdbef14dafe3dd9f1f38cec0ba60ea63a236f46ab1b"
Nov 03 19:45:13.213404 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:13.213367    5591 generic.go:356] "Generic (PLEG): container finished" podID="fd31e715-d9ba-4470-8550-3381c232070d" containerID="6e872647bda75185600bf15a66977672777224e3a63e7eff9bb87822785fd773" exitCode=60
Nov 03 19:45:13.213816 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:13.213452    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-wsk95" event={"ID":"fd31e715-d9ba-4470-8550-3381c232070d","Type":"ContainerDied","Data":"6e872647bda75185600bf15a66977672777224e3a63e7eff9bb87822785fd773"}
Nov 03 19:45:13.213816 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:13.213518    5591 scope.go:122] "RemoveContainer" containerID="b79a48061b327e92a42f5cdbef14dafe3dd9f1f38cec0ba60ea63a236f46ab1b"
Nov 03 19:45:13.213816 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:13.213764    5591 scope.go:122] "RemoveContainer" containerID="6e872647bda75185600bf15a66977672777224e3a63e7eff9bb87822785fd773"
Nov 03 19:45:13.214043 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:45:13.214025    5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895585-wsk95_odh-ai-gateway-infra(fd31e715-d9ba-4470-8550-3381c232070d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-wsk95" podUID="fd31e715-d9ba-4470-8550-3381c232070d"
Nov 03 19:45:14.226027 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:14.225993    5591 scope.go:122] "RemoveContainer" containerID="6e872647bda75185600bf15a66977672777224e3a63e7eff9bb87822785fd773"
Nov 03 19:45:14.226455 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:45:14.226248    5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895585-wsk95_odh-ai-gateway-infra(fd31e715-d9ba-4470-8550-3381c232070d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-wsk95" podUID="fd31e715-d9ba-4470-8550-3381c232070d"
Nov 03 19:45:23.598943 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:23.598907    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-wn8zn"]
Nov 03 19:45:23.603318 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:23.603302    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-wn8zn"
Nov 03 19:45:23.606971 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:23.606948    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 03 19:45:23.607857 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:23.607840    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 03 19:45:23.607924 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:23.607907    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-86jr6\""
Nov 03 19:45:23.611608 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:23.611586    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-wn8zn"]
Nov 03 19:45:23.702988 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:23.702943    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92txt\" (UniqueName: \"kubernetes.io/projected/54e99d0c-3775-47a1-b523-abed9239b7b1-kube-api-access-92txt\") pod \"ai-gateway-controller-5d8dddb676-wn8zn\" (UID: \"54e99d0c-3775-47a1-b523-abed9239b7b1\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-wn8zn"
Nov 03 19:45:23.702988 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:23.702995    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54e99d0c-3775-47a1-b523-abed9239b7b1-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-wn8zn\" (UID: \"54e99d0c-3775-47a1-b523-abed9239b7b1\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-wn8zn"
Nov 03 19:45:23.803659 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:23.803627    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54e99d0c-3775-47a1-b523-abed9239b7b1-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-wn8zn\" (UID: \"54e99d0c-3775-47a1-b523-abed9239b7b1\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-wn8zn"
Nov 03 19:45:23.803856 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:23.803704    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92txt\" (UniqueName: \"kubernetes.io/projected/54e99d0c-3775-47a1-b523-abed9239b7b1-kube-api-access-92txt\") pod \"ai-gateway-controller-5d8dddb676-wn8zn\" (UID: \"54e99d0c-3775-47a1-b523-abed9239b7b1\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-wn8zn"
Nov 03 19:45:23.806284 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:23.806258    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54e99d0c-3775-47a1-b523-abed9239b7b1-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-wn8zn\" (UID: \"54e99d0c-3775-47a1-b523-abed9239b7b1\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-wn8zn"
Nov 03 19:45:23.816482 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:23.816456    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92txt\" (UniqueName: \"kubernetes.io/projected/54e99d0c-3775-47a1-b523-abed9239b7b1-kube-api-access-92txt\") pod \"ai-gateway-controller-5d8dddb676-wn8zn\" (UID: \"54e99d0c-3775-47a1-b523-abed9239b7b1\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-wn8zn"
Nov 03 19:45:23.916900 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:23.916808    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-wn8zn"
Nov 03 19:45:24.190451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:24.190381    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-wn8zn"]
Nov 03 19:45:24.193347 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:45:24.193317    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e99d0c_3775_47a1_b523_abed9239b7b1.slice/crio-16fa30a51614276e01aeb11afe1f6eded77faeb39c758d2015c87b1fd85f7461 WatchSource:0}: Error finding container 16fa30a51614276e01aeb11afe1f6eded77faeb39c758d2015c87b1fd85f7461: Status 404 returned error can't find the container with id 16fa30a51614276e01aeb11afe1f6eded77faeb39c758d2015c87b1fd85f7461
Nov 03 19:45:24.269653 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:24.269619    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-wn8zn" event={"ID":"54e99d0c-3775-47a1-b523-abed9239b7b1","Type":"ContainerStarted","Data":"16fa30a51614276e01aeb11afe1f6eded77faeb39c758d2015c87b1fd85f7461"}
Nov 03 19:45:27.341741 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:27.341707    5591 scope.go:122] "RemoveContainer" containerID="6e872647bda75185600bf15a66977672777224e3a63e7eff9bb87822785fd773"
Nov 03 19:45:28.253190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.253161    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-45rfv"]
Nov 03 19:45:28.256588 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.256571    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-45rfv"
Nov 03 19:45:28.260924 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.260865    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 03 19:45:28.261956 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.261927    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-2tlvj\""
Nov 03 19:45:28.265892 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.265868    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-45rfv"]
Nov 03 19:45:28.292125 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.292094    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-wn8zn" event={"ID":"54e99d0c-3775-47a1-b523-abed9239b7b1","Type":"ContainerStarted","Data":"5674b835af4c9c55a6164dacef5116040de2a386e4b800e7db9e643f66f9994f"}
Nov 03 19:45:28.292278 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.292154    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-wn8zn"
Nov 03 19:45:28.294532 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.294507    5591 generic.go:356] "Generic (PLEG): container finished" podID="fd31e715-d9ba-4470-8550-3381c232070d" containerID="3929b18ddcfc9e314891bb702d4984575e7b48b4816becad630d6674470b2c8b" exitCode=60
Nov 03 19:45:28.294724 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.294615    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-wsk95" event={"ID":"fd31e715-d9ba-4470-8550-3381c232070d","Type":"ContainerDied","Data":"3929b18ddcfc9e314891bb702d4984575e7b48b4816becad630d6674470b2c8b"}
Nov 03 19:45:28.294833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.294761    5591 scope.go:122] "RemoveContainer" containerID="6e872647bda75185600bf15a66977672777224e3a63e7eff9bb87822785fd773"
Nov 03 19:45:28.295175 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.295158    5591 scope.go:122] "RemoveContainer" containerID="3929b18ddcfc9e314891bb702d4984575e7b48b4816becad630d6674470b2c8b"
Nov 03 19:45:28.295442 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:45:28.295418    5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895585-wsk95_odh-ai-gateway-infra(fd31e715-d9ba-4470-8550-3381c232070d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-wsk95" podUID="fd31e715-d9ba-4470-8550-3381c232070d"
Nov 03 19:45:28.295551 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.295533    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-x84lg"]
Nov 03 19:45:28.299672 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.299647    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-x84lg"
Nov 03 19:45:28.310069 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.310040    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-x84lg"]
Nov 03 19:45:28.312328 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.312247    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-wn8zn" podStartSLOduration=2.001581787 podStartE2EDuration="5.312232689s" totalImagesPullingTime="3.310650902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:28.311939231 +0000 UTC m=+477.084232250" watchObservedRunningTime="2026-11-03 19:45:28.312232689 +0000 UTC m=+477.084525708"
Nov 03 19:45:28.344990 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.344952    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc67cda2-1534-4223-be48-0130220d6360-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-45rfv\" (UID: \"fc67cda2-1534-4223-be48-0130220d6360\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-45rfv"
Nov 03 19:45:28.345443 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.345081    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d-plugins-config-volume\") pod \"payload-processing-7bb9489676-x84lg\" (UID: \"d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d\") " pod="openshift-ingress/payload-processing-7bb9489676-x84lg"
Nov 03 19:45:28.345443 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.345281    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr967\" (UniqueName: \"kubernetes.io/projected/fc67cda2-1534-4223-be48-0130220d6360-kube-api-access-jr967\") pod \"payload-pre-processing-7b98bb678c-45rfv\" (UID: \"fc67cda2-1534-4223-be48-0130220d6360\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-45rfv"
Nov 03 19:45:28.345443 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.345367    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jsgj\" (UniqueName: \"kubernetes.io/projected/d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d-kube-api-access-8jsgj\") pod \"payload-processing-7bb9489676-x84lg\" (UID: \"d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d\") " pod="openshift-ingress/payload-processing-7bb9489676-x84lg"
Nov 03 19:45:28.446577 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.446494    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc67cda2-1534-4223-be48-0130220d6360-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-45rfv\" (UID: \"fc67cda2-1534-4223-be48-0130220d6360\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-45rfv"
Nov 03 19:45:28.446577 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.446539    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d-plugins-config-volume\") pod \"payload-processing-7bb9489676-x84lg\" (UID: \"d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d\") " pod="openshift-ingress/payload-processing-7bb9489676-x84lg"
Nov 03 19:45:28.446577 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.446570    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr967\" (UniqueName: \"kubernetes.io/projected/fc67cda2-1534-4223-be48-0130220d6360-kube-api-access-jr967\") pod \"payload-pre-processing-7b98bb678c-45rfv\" (UID: \"fc67cda2-1534-4223-be48-0130220d6360\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-45rfv"
Nov 03 19:45:28.446861 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.446603    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jsgj\" (UniqueName: \"kubernetes.io/projected/d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d-kube-api-access-8jsgj\") pod \"payload-processing-7bb9489676-x84lg\" (UID: \"d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d\") " pod="openshift-ingress/payload-processing-7bb9489676-x84lg"
Nov 03 19:45:28.447351 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.447327    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc67cda2-1534-4223-be48-0130220d6360-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-45rfv\" (UID: \"fc67cda2-1534-4223-be48-0130220d6360\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-45rfv"
Nov 03 19:45:28.447351 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.447343    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d-plugins-config-volume\") pod \"payload-processing-7bb9489676-x84lg\" (UID: \"d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d\") " pod="openshift-ingress/payload-processing-7bb9489676-x84lg"
Nov 03 19:45:28.459160 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.459132    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jsgj\" (UniqueName: \"kubernetes.io/projected/d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d-kube-api-access-8jsgj\") pod \"payload-processing-7bb9489676-x84lg\" (UID: \"d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d\") " pod="openshift-ingress/payload-processing-7bb9489676-x84lg"
Nov 03 19:45:28.459397 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.459374    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr967\" (UniqueName: \"kubernetes.io/projected/fc67cda2-1534-4223-be48-0130220d6360-kube-api-access-jr967\") pod \"payload-pre-processing-7b98bb678c-45rfv\" (UID: \"fc67cda2-1534-4223-be48-0130220d6360\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-45rfv"
Nov 03 19:45:28.570028 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.569990    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-45rfv"
Nov 03 19:45:28.617740 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.617697    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-x84lg"
Nov 03 19:45:28.725895 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:45:28.725811    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:45:28.844755 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.844613    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-45rfv"]
Nov 03 19:45:28.847358 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:45:28.847334    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc67cda2_1534_4223_be48_0130220d6360.slice/crio-9e205e75018541a355d7387af293a560e32e39053879db453a48b4abc862837c WatchSource:0}: Error finding container 9e205e75018541a355d7387af293a560e32e39053879db453a48b4abc862837c: Status 404 returned error can't find the container with id 9e205e75018541a355d7387af293a560e32e39053879db453a48b4abc862837c
Nov 03 19:45:28.902339 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:28.902314    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-x84lg"]
Nov 03 19:45:28.904509 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:45:28.904482    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2672d13_b0a8_4e7e_81f9_4b0a04e69e1d.slice/crio-adf51a1c316450a9a00e62c7026d053ae5588b54a13ab43fcd72146a4c5ca6d7 WatchSource:0}: Error finding container adf51a1c316450a9a00e62c7026d053ae5588b54a13ab43fcd72146a4c5ca6d7: Status 404 returned error can't find the container with id adf51a1c316450a9a00e62c7026d053ae5588b54a13ab43fcd72146a4c5ca6d7
Nov 03 19:45:29.300740 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:29.300697    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-x84lg" event={"ID":"d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d","Type":"ContainerStarted","Data":"adf51a1c316450a9a00e62c7026d053ae5588b54a13ab43fcd72146a4c5ca6d7"}
Nov 03 19:45:29.302067 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:29.302040    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-45rfv" event={"ID":"fc67cda2-1534-4223-be48-0130220d6360","Type":"ContainerStarted","Data":"9e205e75018541a355d7387af293a560e32e39053879db453a48b4abc862837c"}
Nov 03 19:45:29.348223 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:29.348182    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-wsk95"]
Nov 03 19:45:29.534226 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:29.534171    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-wsk95"
Nov 03 19:45:29.558505 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:29.557974    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd31e715-d9ba-4470-8550-3381c232070d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd31e715-d9ba-4470-8550-3381c232070d-maas-api-service-ca\") pod \"fd31e715-d9ba-4470-8550-3381c232070d\" (UID: \"fd31e715-d9ba-4470-8550-3381c232070d\") "
Nov 03 19:45:29.558505 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:29.558043    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd31e715-d9ba-4470-8550-3381c232070d-kube-api-access-vbfrn\" (UniqueName: \"kubernetes.io/projected/fd31e715-d9ba-4470-8550-3381c232070d-kube-api-access-vbfrn\") pod \"fd31e715-d9ba-4470-8550-3381c232070d\" (UID: \"fd31e715-d9ba-4470-8550-3381c232070d\") "
Nov 03 19:45:29.560236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:29.558922    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd31e715-d9ba-4470-8550-3381c232070d-maas-api-service-ca" pod "fd31e715-d9ba-4470-8550-3381c232070d" (UID: "fd31e715-d9ba-4470-8550-3381c232070d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:45:29.574401 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:29.574368    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd31e715-d9ba-4470-8550-3381c232070d-kube-api-access-vbfrn" pod "fd31e715-d9ba-4470-8550-3381c232070d" (UID: "fd31e715-d9ba-4470-8550-3381c232070d"). InnerVolumeSpecName "kube-api-access-vbfrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:29.659147 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:29.659112    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbfrn\" (UniqueName: \"kubernetes.io/projected/fd31e715-d9ba-4470-8550-3381c232070d-kube-api-access-vbfrn\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:29.659147 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:29.659148    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd31e715-d9ba-4470-8550-3381c232070d-maas-api-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:30.311988 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:30.311948    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-wsk95" event={"ID":"fd31e715-d9ba-4470-8550-3381c232070d","Type":"ContainerDied","Data":"c8e7a9107414098a037abbe05a73d70d3897e11201c34e67bf558be375a0573d"}
Nov 03 19:45:30.312149 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:30.312027    5591 scope.go:122] "RemoveContainer" containerID="3929b18ddcfc9e314891bb702d4984575e7b48b4816becad630d6674470b2c8b"
Nov 03 19:45:30.312149 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:30.312130    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-wsk95"
Nov 03 19:45:30.337385 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:30.337357    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-wsk95"]
Nov 03 19:45:30.349670 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:30.343341    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-wsk95"]
Nov 03 19:45:31.319397 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:31.319370    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-45rfv" event={"ID":"fc67cda2-1534-4223-be48-0130220d6360","Type":"ContainerStarted","Data":"74e343698c5915fb2a6daf91df1dbe22c16e3b6fc5f2f4918b78c0d48ff099e4"}
Nov 03 19:45:31.319573 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:31.319516    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-45rfv"
Nov 03 19:45:31.338653 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:31.338597    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-45rfv" podStartSLOduration=1.011730676 podStartE2EDuration="3.338577999s" totalImagesPullingTime="2.326847323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:31.338134881 +0000 UTC m=+480.110427903" watchObservedRunningTime="2026-11-03 19:45:31.338577999 +0000 UTC m=+480.110871019"
Nov 03 19:45:31.346654 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:31.346628    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd31e715-d9ba-4470-8550-3381c232070d" path="/var/lib/kubelet/pods/fd31e715-d9ba-4470-8550-3381c232070d/volumes"
Nov 03 19:45:32.325475 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:32.325435    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-x84lg" event={"ID":"d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d","Type":"ContainerStarted","Data":"9311d981c99f20f4a4988aebc309b5ab0ffb2ca28a0c511af908f2ac3d7cba0f"}
Nov 03 19:45:32.325890 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:32.325568    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7bb9489676-x84lg"
Nov 03 19:45:32.345080 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:32.345031    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7bb9489676-x84lg" podStartSLOduration=1.9721729099999998 podStartE2EDuration="4.345016967s" totalImagesPullingTime="2.372844057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:32.343819391 +0000 UTC m=+481.116112410" watchObservedRunningTime="2026-11-03 19:45:32.345016967 +0000 UTC m=+481.117309987"
Nov 03 19:45:39.305977 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:39.305943    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-wn8zn"
Nov 03 19:45:39.577299 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:39.577218    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-79b8b796f-2sshp"]
Nov 03 19:45:39.577579 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:39.577567    5591 cpu_manager.go:414] "containerMap: removing container" podUID="fd31e715-d9ba-4470-8550-3381c232070d" containerName="cleanup"
Nov 03 19:45:39.577625 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:39.577581    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd31e715-d9ba-4470-8550-3381c232070d" containerName="cleanup"
Nov 03 19:45:39.577625 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:39.577613    5591 cpu_manager.go:414] "containerMap: removing container" podUID="fd31e715-d9ba-4470-8550-3381c232070d" containerName="cleanup"
Nov 03 19:45:39.577625 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:39.577618    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd31e715-d9ba-4470-8550-3381c232070d" containerName="cleanup"
Nov 03 19:45:39.577723 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:39.577674    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd31e715-d9ba-4470-8550-3381c232070d" containerName="cleanup"
Nov 03 19:45:39.577723 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:39.577686    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd31e715-d9ba-4470-8550-3381c232070d" containerName="cleanup"
Nov 03 19:45:39.577723 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:39.577691    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd31e715-d9ba-4470-8550-3381c232070d" containerName="cleanup"
Nov 03 19:45:39.581949 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:39.581925    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-79b8b796f-2sshp"
Nov 03 19:45:39.591407 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:39.591378    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-79b8b796f-2sshp"]
Nov 03 19:45:39.651081 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:39.651050    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/747740b4-92e0-4179-9318-04fd9030b8e0-metrics-certs\") pod \"ai-gateway-controller-79b8b796f-2sshp\" (UID: \"747740b4-92e0-4179-9318-04fd9030b8e0\") " pod="opendatahub/ai-gateway-controller-79b8b796f-2sshp"
Nov 03 19:45:39.651297 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:39.651124    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2zgp\" (UniqueName: \"kubernetes.io/projected/747740b4-92e0-4179-9318-04fd9030b8e0-kube-api-access-h2zgp\") pod \"ai-gateway-controller-79b8b796f-2sshp\" (UID: \"747740b4-92e0-4179-9318-04fd9030b8e0\") " pod="opendatahub/ai-gateway-controller-79b8b796f-2sshp"
Nov 03 19:45:39.751979 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:39.751940    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2zgp\" (UniqueName: \"kubernetes.io/projected/747740b4-92e0-4179-9318-04fd9030b8e0-kube-api-access-h2zgp\") pod \"ai-gateway-controller-79b8b796f-2sshp\" (UID: \"747740b4-92e0-4179-9318-04fd9030b8e0\") " pod="opendatahub/ai-gateway-controller-79b8b796f-2sshp"
Nov 03 19:45:39.752151 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:39.751997    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/747740b4-92e0-4179-9318-04fd9030b8e0-metrics-certs\") pod \"ai-gateway-controller-79b8b796f-2sshp\" (UID: \"747740b4-92e0-4179-9318-04fd9030b8e0\") " pod="opendatahub/ai-gateway-controller-79b8b796f-2sshp"
Nov 03 19:45:39.754540 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:39.754518    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/747740b4-92e0-4179-9318-04fd9030b8e0-metrics-certs\") pod \"ai-gateway-controller-79b8b796f-2sshp\" (UID: \"747740b4-92e0-4179-9318-04fd9030b8e0\") " pod="opendatahub/ai-gateway-controller-79b8b796f-2sshp"
Nov 03 19:45:39.765487 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:39.765464    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2zgp\" (UniqueName: \"kubernetes.io/projected/747740b4-92e0-4179-9318-04fd9030b8e0-kube-api-access-h2zgp\") pod \"ai-gateway-controller-79b8b796f-2sshp\" (UID: \"747740b4-92e0-4179-9318-04fd9030b8e0\") " pod="opendatahub/ai-gateway-controller-79b8b796f-2sshp"
Nov 03 19:45:39.895722 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:39.895630    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-79b8b796f-2sshp"
Nov 03 19:45:40.170997 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:40.170923    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-79b8b796f-2sshp"]
Nov 03 19:45:40.173978 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:45:40.173943    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747740b4_92e0_4179_9318_04fd9030b8e0.slice/crio-8ab7a09ccb8ed8269a7602a5f08326523d0151cbf052483fbdc2643d1a6f634e WatchSource:0}: Error finding container 8ab7a09ccb8ed8269a7602a5f08326523d0151cbf052483fbdc2643d1a6f634e: Status 404 returned error can't find the container with id 8ab7a09ccb8ed8269a7602a5f08326523d0151cbf052483fbdc2643d1a6f634e
Nov 03 19:45:40.361350 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:40.361298    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-79b8b796f-2sshp" event={"ID":"747740b4-92e0-4179-9318-04fd9030b8e0","Type":"ContainerStarted","Data":"8ab7a09ccb8ed8269a7602a5f08326523d0151cbf052483fbdc2643d1a6f634e"}
Nov 03 19:45:41.366511 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:41.366470    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-79b8b796f-2sshp" event={"ID":"747740b4-92e0-4179-9318-04fd9030b8e0","Type":"ContainerStarted","Data":"ad292d28088cdbc51c5564dd3455e4af7fcec8ed4426d766346b7b8709812843"}
Nov 03 19:45:41.366960 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:41.366561    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-79b8b796f-2sshp"
Nov 03 19:45:41.386401 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:41.386294    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-79b8b796f-2sshp" podStartSLOduration=1.935210325 podStartE2EDuration="2.386277981s" totalImagesPullingTime="451.067656ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:41.385796929 +0000 UTC m=+490.158089949" watchObservedRunningTime="2026-11-03 19:45:41.386277981 +0000 UTC m=+490.158571000"
Nov 03 19:45:42.329133 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:42.329096    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-45rfv"
Nov 03 19:45:43.332450 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:43.332423    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7bb9489676-x84lg"
Nov 03 19:45:52.373578 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:52.373551    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-79b8b796f-2sshp"
Nov 03 19:45:52.421892 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:52.421859    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-wn8zn"]
Nov 03 19:45:52.422106 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:52.422082    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-wn8zn" podUID="54e99d0c-3775-47a1-b523-abed9239b7b1" containerName="manager" containerID="cri-o://5674b835af4c9c55a6164dacef5116040de2a386e4b800e7db9e643f66f9994f" gracePeriod=30
Nov 03 19:45:52.669077 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:52.669056    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-wn8zn"
Nov 03 19:45:52.763645 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:52.763613    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54e99d0c-3775-47a1-b523-abed9239b7b1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/54e99d0c-3775-47a1-b523-abed9239b7b1-metrics-certs\") pod \"54e99d0c-3775-47a1-b523-abed9239b7b1\" (UID: \"54e99d0c-3775-47a1-b523-abed9239b7b1\") "
Nov 03 19:45:52.763811 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:52.763655    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54e99d0c-3775-47a1-b523-abed9239b7b1-kube-api-access-92txt\" (UniqueName: \"kubernetes.io/projected/54e99d0c-3775-47a1-b523-abed9239b7b1-kube-api-access-92txt\") pod \"54e99d0c-3775-47a1-b523-abed9239b7b1\" (UID: \"54e99d0c-3775-47a1-b523-abed9239b7b1\") "
Nov 03 19:45:52.765762 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:52.765730    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54e99d0c-3775-47a1-b523-abed9239b7b1-metrics-certs" pod "54e99d0c-3775-47a1-b523-abed9239b7b1" (UID: "54e99d0c-3775-47a1-b523-abed9239b7b1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:45:52.765871 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:52.765835    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54e99d0c-3775-47a1-b523-abed9239b7b1-kube-api-access-92txt" pod "54e99d0c-3775-47a1-b523-abed9239b7b1" (UID: "54e99d0c-3775-47a1-b523-abed9239b7b1"). InnerVolumeSpecName "kube-api-access-92txt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:52.864771 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:52.864690    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54e99d0c-3775-47a1-b523-abed9239b7b1-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:52.864771 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:52.864717    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92txt\" (UniqueName: \"kubernetes.io/projected/54e99d0c-3775-47a1-b523-abed9239b7b1-kube-api-access-92txt\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:53.421097 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:53.421061    5591 generic.go:356] "Generic (PLEG): container finished" podID="54e99d0c-3775-47a1-b523-abed9239b7b1" containerID="5674b835af4c9c55a6164dacef5116040de2a386e4b800e7db9e643f66f9994f" exitCode=0
Nov 03 19:45:53.421568 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:53.421127    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-wn8zn"
Nov 03 19:45:53.421568 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:53.421141    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-wn8zn" event={"ID":"54e99d0c-3775-47a1-b523-abed9239b7b1","Type":"ContainerDied","Data":"5674b835af4c9c55a6164dacef5116040de2a386e4b800e7db9e643f66f9994f"}
Nov 03 19:45:53.421568 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:53.421194    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-wn8zn" event={"ID":"54e99d0c-3775-47a1-b523-abed9239b7b1","Type":"ContainerDied","Data":"16fa30a51614276e01aeb11afe1f6eded77faeb39c758d2015c87b1fd85f7461"}
Nov 03 19:45:53.421568 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:53.421232    5591 scope.go:122] "RemoveContainer" containerID="5674b835af4c9c55a6164dacef5116040de2a386e4b800e7db9e643f66f9994f"
Nov 03 19:45:53.430095 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:53.429914    5591 scope.go:122] "RemoveContainer" containerID="5674b835af4c9c55a6164dacef5116040de2a386e4b800e7db9e643f66f9994f"
Nov 03 19:45:53.430238 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:45:53.430213    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5674b835af4c9c55a6164dacef5116040de2a386e4b800e7db9e643f66f9994f\": container with ID starting with 5674b835af4c9c55a6164dacef5116040de2a386e4b800e7db9e643f66f9994f not found: ID does not exist" containerID="5674b835af4c9c55a6164dacef5116040de2a386e4b800e7db9e643f66f9994f"
Nov 03 19:45:53.430307 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:53.430246    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5674b835af4c9c55a6164dacef5116040de2a386e4b800e7db9e643f66f9994f"} err="failed to get container status \"5674b835af4c9c55a6164dacef5116040de2a386e4b800e7db9e643f66f9994f\": rpc error: code = NotFound desc = could not find container \"5674b835af4c9c55a6164dacef5116040de2a386e4b800e7db9e643f66f9994f\": container with ID starting with 5674b835af4c9c55a6164dacef5116040de2a386e4b800e7db9e643f66f9994f not found: ID does not exist"
Nov 03 19:45:53.438647 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:53.438626    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-wn8zn"]
Nov 03 19:45:53.442842 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:53.442822    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-wn8zn"]
Nov 03 19:45:55.346869 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:45:55.346839    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54e99d0c-3775-47a1-b523-abed9239b7b1" path="/var/lib/kubelet/pods/54e99d0c-3775-47a1-b523-abed9239b7b1/volumes"
Nov 03 19:45:58.721982 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:45:58.721949    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:46:08.992865 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:08.992835    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg"]
Nov 03 19:46:08.993342 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:08.993245    5591 cpu_manager.go:414] "containerMap: removing container" podUID="fd31e715-d9ba-4470-8550-3381c232070d" containerName="cleanup"
Nov 03 19:46:08.993342 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:08.993258    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd31e715-d9ba-4470-8550-3381c232070d" containerName="cleanup"
Nov 03 19:46:08.993342 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:08.993268    5591 cpu_manager.go:414] "containerMap: removing container" podUID="54e99d0c-3775-47a1-b523-abed9239b7b1" containerName="manager"
Nov 03 19:46:08.993342 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:08.993275    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54e99d0c-3775-47a1-b523-abed9239b7b1" containerName="manager"
Nov 03 19:46:08.993342 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:08.993332    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="54e99d0c-3775-47a1-b523-abed9239b7b1" containerName="manager"
Nov 03 19:46:09.001068 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:09.001048    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg"
Nov 03 19:46:09.009456 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:09.009432    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg"]
Nov 03 19:46:09.012606 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:09.012578    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8tzmp\""
Nov 03 19:46:09.017163 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:09.017146    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 19:46:09.017463 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:09.017441    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 19:46:09.109076 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:09.109045    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc960ce4-0839-4aa4-b30c-c64f550b1655-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg\" (UID: \"dc960ce4-0839-4aa4-b30c-c64f550b1655\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg"
Nov 03 19:46:09.109253 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:09.109094    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc960ce4-0839-4aa4-b30c-c64f550b1655-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg\" (UID: \"dc960ce4-0839-4aa4-b30c-c64f550b1655\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg"
Nov 03 19:46:09.109253 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:09.109116    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27jww\" (UniqueName: \"kubernetes.io/projected/dc960ce4-0839-4aa4-b30c-c64f550b1655-kube-api-access-27jww\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg\" (UID: \"dc960ce4-0839-4aa4-b30c-c64f550b1655\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg"
Nov 03 19:46:09.109333 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:09.109299    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc960ce4-0839-4aa4-b30c-c64f550b1655-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg\" (UID: \"dc960ce4-0839-4aa4-b30c-c64f550b1655\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg"
Nov 03 19:46:09.109367 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:09.109330    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc960ce4-0839-4aa4-b30c-c64f550b1655-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg\" (UID: \"dc960ce4-0839-4aa4-b30c-c64f550b1655\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg"
Nov 03 19:46:09.109367 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:09.109355    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc960ce4-0839-4aa4-b30c-c64f550b1655-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg\" (UID: \"dc960ce4-0839-4aa4-b30c-c64f550b1655\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg"
Nov 03 19:46:09.119759 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:09.119735    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 03 19:46:09.210163 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:09.210126    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc960ce4-0839-4aa4-b30c-c64f550b1655-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg\" (UID: \"dc960ce4-0839-4aa4-b30c-c64f550b1655\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg"
Nov 03 19:46:09.210163 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:09.210164    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27jww\" (UniqueName: \"kubernetes.io/projected/dc960ce4-0839-4aa4-b30c-c64f550b1655-kube-api-access-27jww\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg\" (UID: \"dc960ce4-0839-4aa4-b30c-c64f550b1655\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg"
Nov 03 19:46:09.210423 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:09.210232    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc960ce4-0839-4aa4-b30c-c64f550b1655-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg\" (UID: \"dc960ce4-0839-4aa4-b30c-c64f550b1655\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg"
Nov 03 19:46:09.210423 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:09.210268    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc960ce4-0839-4aa4-b30c-c64f550b1655-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg\" (UID: \"dc960ce4-0839-4aa4-b30c-c64f550b1655\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg"
Nov 03 19:46:09.210423 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:09.210296    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc960ce4-0839-4aa4-b30c-c64f550b1655-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg\" (UID: \"dc960ce4-0839-4aa4-b30c-c64f550b1655\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg"
Nov 03 19:46:09.210423 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:09.210359    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc960ce4-0839-4aa4-b30c-c64f550b1655-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg\" (UID: \"dc960ce4-0839-4aa4-b30c-c64f550b1655\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg"
Nov 03 19:46:09.210705 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:09.210684    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc960ce4-0839-4aa4-b30c-c64f550b1655-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg\" (UID: \"dc960ce4-0839-4aa4-b30c-c64f550b1655\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg"
Nov 03 19:46:09.210759 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:09.210719    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc960ce4-0839-4aa4-b30c-c64f550b1655-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg\" (UID: \"dc960ce4-0839-4aa4-b30c-c64f550b1655\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg"
Nov 03 19:46:09.210759 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:09.210747    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc960ce4-0839-4aa4-b30c-c64f550b1655-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg\" (UID: \"dc960ce4-0839-4aa4-b30c-c64f550b1655\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg"
Nov 03 19:46:09.212667 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:09.212638    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc960ce4-0839-4aa4-b30c-c64f550b1655-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg\" (UID: \"dc960ce4-0839-4aa4-b30c-c64f550b1655\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg"
Nov 03 19:46:09.213016 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:09.212995    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc960ce4-0839-4aa4-b30c-c64f550b1655-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg\" (UID: \"dc960ce4-0839-4aa4-b30c-c64f550b1655\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg"
Nov 03 19:46:09.222746 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:09.222726    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27jww\" (UniqueName: \"kubernetes.io/projected/dc960ce4-0839-4aa4-b30c-c64f550b1655-kube-api-access-27jww\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg\" (UID: \"dc960ce4-0839-4aa4-b30c-c64f550b1655\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg"
Nov 03 19:46:09.314340 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:09.314312    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg"
Nov 03 19:46:09.548470 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:09.548432    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:46:09.591490 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:09.591419    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg"]
Nov 03 19:46:09.597714 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:46:09.597686    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc960ce4_0839_4aa4_b30c_c64f550b1655.slice/crio-4982f9e8d56225d33be7f8bb6e0a6393013d189b32abb2a456bc59ee0d1d8ef1 WatchSource:0}: Error finding container 4982f9e8d56225d33be7f8bb6e0a6393013d189b32abb2a456bc59ee0d1d8ef1: Status 404 returned error can't find the container with id 4982f9e8d56225d33be7f8bb6e0a6393013d189b32abb2a456bc59ee0d1d8ef1
Nov 03 19:46:10.498841 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:10.498802    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg" event={"ID":"dc960ce4-0839-4aa4-b30c-c64f550b1655","Type":"ContainerStarted","Data":"4982f9e8d56225d33be7f8bb6e0a6393013d189b32abb2a456bc59ee0d1d8ef1"}
Nov 03 19:46:11.504609 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:11.504532    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg" event={"ID":"dc960ce4-0839-4aa4-b30c-c64f550b1655","Type":"ContainerStarted","Data":"d15555c1d104f44362798aa62129ffff370e63c70df9f01f9354808f44bd5439"}
Nov 03 19:46:11.525623 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:11.525574    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg" podStartSLOduration=2.092533674 podStartE2EDuration="3.525560425s" totalImagesPullingTime="1.433026751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:11.524025826 +0000 UTC m=+520.296318843" watchObservedRunningTime="2026-11-03 19:46:11.525560425 +0000 UTC m=+520.297853444"
Nov 03 19:46:12.315172 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:12.315139    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg"
Nov 03 19:46:13.087574 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:13.087540    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:46:13.194551 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:13.194523    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm"]
Nov 03 19:46:13.198087 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:13.198067    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm"
Nov 03 19:46:13.206116 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:13.206088    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm"]
Nov 03 19:46:13.207429 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:13.207402    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 03 19:46:13.245811 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:13.245782    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0de7df58-cc36-454f-8954-fb6cb562da6a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2fbm\" (UID: \"0de7df58-cc36-454f-8954-fb6cb562da6a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm"
Nov 03 19:46:13.245985 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:13.245820    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0de7df58-cc36-454f-8954-fb6cb562da6a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2fbm\" (UID: \"0de7df58-cc36-454f-8954-fb6cb562da6a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm"
Nov 03 19:46:13.245985 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:13.245839    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0de7df58-cc36-454f-8954-fb6cb562da6a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2fbm\" (UID: \"0de7df58-cc36-454f-8954-fb6cb562da6a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm"
Nov 03 19:46:13.245985 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:13.245854    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0de7df58-cc36-454f-8954-fb6cb562da6a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2fbm\" (UID: \"0de7df58-cc36-454f-8954-fb6cb562da6a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm"
Nov 03 19:46:13.245985 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:13.245966    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0de7df58-cc36-454f-8954-fb6cb562da6a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2fbm\" (UID: \"0de7df58-cc36-454f-8954-fb6cb562da6a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm"
Nov 03 19:46:13.246131 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:13.246009    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z884r\" (UniqueName: \"kubernetes.io/projected/0de7df58-cc36-454f-8954-fb6cb562da6a-kube-api-access-z884r\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2fbm\" (UID: \"0de7df58-cc36-454f-8954-fb6cb562da6a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm"
Nov 03 19:46:13.347164 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:13.347091    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0de7df58-cc36-454f-8954-fb6cb562da6a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2fbm\" (UID: \"0de7df58-cc36-454f-8954-fb6cb562da6a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm"
Nov 03 19:46:13.347164 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:13.347129    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0de7df58-cc36-454f-8954-fb6cb562da6a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2fbm\" (UID: \"0de7df58-cc36-454f-8954-fb6cb562da6a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm"
Nov 03 19:46:13.347164 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:13.347144    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0de7df58-cc36-454f-8954-fb6cb562da6a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2fbm\" (UID: \"0de7df58-cc36-454f-8954-fb6cb562da6a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm"
Nov 03 19:46:13.347461 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:13.347294    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0de7df58-cc36-454f-8954-fb6cb562da6a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2fbm\" (UID: \"0de7df58-cc36-454f-8954-fb6cb562da6a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm"
Nov 03 19:46:13.347461 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:13.347360    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0de7df58-cc36-454f-8954-fb6cb562da6a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2fbm\" (UID: \"0de7df58-cc36-454f-8954-fb6cb562da6a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm"
Nov 03 19:46:13.347461 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:13.347410    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z884r\" (UniqueName: \"kubernetes.io/projected/0de7df58-cc36-454f-8954-fb6cb562da6a-kube-api-access-z884r\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2fbm\" (UID: \"0de7df58-cc36-454f-8954-fb6cb562da6a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm"
Nov 03 19:46:13.347656 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:13.347637    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0de7df58-cc36-454f-8954-fb6cb562da6a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2fbm\" (UID: \"0de7df58-cc36-454f-8954-fb6cb562da6a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm"
Nov 03 19:46:13.347718 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:13.347667    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0de7df58-cc36-454f-8954-fb6cb562da6a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2fbm\" (UID: \"0de7df58-cc36-454f-8954-fb6cb562da6a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm"
Nov 03 19:46:13.347770 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:13.347739    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0de7df58-cc36-454f-8954-fb6cb562da6a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2fbm\" (UID: \"0de7df58-cc36-454f-8954-fb6cb562da6a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm"
Nov 03 19:46:13.349800 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:13.349779    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0de7df58-cc36-454f-8954-fb6cb562da6a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2fbm\" (UID: \"0de7df58-cc36-454f-8954-fb6cb562da6a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm"
Nov 03 19:46:13.349930 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:13.349812    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0de7df58-cc36-454f-8954-fb6cb562da6a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2fbm\" (UID: \"0de7df58-cc36-454f-8954-fb6cb562da6a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm"
Nov 03 19:46:13.362021 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:13.361986    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z884r\" (UniqueName: \"kubernetes.io/projected/0de7df58-cc36-454f-8954-fb6cb562da6a-kube-api-access-z884r\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-c2fbm\" (UID: \"0de7df58-cc36-454f-8954-fb6cb562da6a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm"
Nov 03 19:46:13.517339 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:13.517312    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm"
Nov 03 19:46:13.791733 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:13.791710    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm"]
Nov 03 19:46:13.794395 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:46:13.794366    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de7df58_cc36_454f_8954_fb6cb562da6a.slice/crio-8e1d04e9aa8cbe124f3ca478709d9e744b73d6050aa22aacd571caac018a4903 WatchSource:0}: Error finding container 8e1d04e9aa8cbe124f3ca478709d9e744b73d6050aa22aacd571caac018a4903: Status 404 returned error can't find the container with id 8e1d04e9aa8cbe124f3ca478709d9e744b73d6050aa22aacd571caac018a4903
Nov 03 19:46:14.425128 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:14.425050    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k"]
Nov 03 19:46:14.428987 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:14.428965    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k"
Nov 03 19:46:14.433546 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:14.433524    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 03 19:46:14.440367 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:14.440345    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k"]
Nov 03 19:46:14.458371 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:14.458337    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn4nc\" (UniqueName: \"kubernetes.io/projected/6efbc03a-f9b7-463d-969e-7c420f0d59e1-kube-api-access-kn4nc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fxm7k\" (UID: \"6efbc03a-f9b7-463d-969e-7c420f0d59e1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k"
Nov 03 19:46:14.458588 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:14.458426    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6efbc03a-f9b7-463d-969e-7c420f0d59e1-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fxm7k\" (UID: \"6efbc03a-f9b7-463d-969e-7c420f0d59e1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k"
Nov 03 19:46:14.458588 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:14.458454    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6efbc03a-f9b7-463d-969e-7c420f0d59e1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fxm7k\" (UID: \"6efbc03a-f9b7-463d-969e-7c420f0d59e1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k"
Nov 03 19:46:14.458710 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:14.458600    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6efbc03a-f9b7-463d-969e-7c420f0d59e1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fxm7k\" (UID: \"6efbc03a-f9b7-463d-969e-7c420f0d59e1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k"
Nov 03 19:46:14.458710 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:14.458650    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6efbc03a-f9b7-463d-969e-7c420f0d59e1-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fxm7k\" (UID: \"6efbc03a-f9b7-463d-969e-7c420f0d59e1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k"
Nov 03 19:46:14.458782 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:14.458728    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6efbc03a-f9b7-463d-969e-7c420f0d59e1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fxm7k\" (UID: \"6efbc03a-f9b7-463d-969e-7c420f0d59e1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k"
Nov 03 19:46:14.519980 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:14.519944    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm" event={"ID":"0de7df58-cc36-454f-8954-fb6cb562da6a","Type":"ContainerStarted","Data":"dff520bbc081385805e7a88f2429c7d5f09e0ad1fb2f27cd1561af254938c5f5"}
Nov 03 19:46:14.520153 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:14.520001    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm" event={"ID":"0de7df58-cc36-454f-8954-fb6cb562da6a","Type":"ContainerStarted","Data":"8e1d04e9aa8cbe124f3ca478709d9e744b73d6050aa22aacd571caac018a4903"}
Nov 03 19:46:14.539891 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:14.539838    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm" podStartSLOduration=1.221595332 podStartE2EDuration="1.539822602s" totalImagesPullingTime="318.22727ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:14.538897709 +0000 UTC m=+523.311190735" watchObservedRunningTime="2026-11-03 19:46:14.539822602 +0000 UTC m=+523.312115619"
Nov 03 19:46:14.560258 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:14.560229    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6efbc03a-f9b7-463d-969e-7c420f0d59e1-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fxm7k\" (UID: \"6efbc03a-f9b7-463d-969e-7c420f0d59e1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k"
Nov 03 19:46:14.560258 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:14.560263    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6efbc03a-f9b7-463d-969e-7c420f0d59e1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fxm7k\" (UID: \"6efbc03a-f9b7-463d-969e-7c420f0d59e1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k"
Nov 03 19:46:14.560500 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:14.560306    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn4nc\" (UniqueName: \"kubernetes.io/projected/6efbc03a-f9b7-463d-969e-7c420f0d59e1-kube-api-access-kn4nc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fxm7k\" (UID: \"6efbc03a-f9b7-463d-969e-7c420f0d59e1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k"
Nov 03 19:46:14.560500 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:14.560344    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6efbc03a-f9b7-463d-969e-7c420f0d59e1-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fxm7k\" (UID: \"6efbc03a-f9b7-463d-969e-7c420f0d59e1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k"
Nov 03 19:46:14.560500 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:14.560365    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6efbc03a-f9b7-463d-969e-7c420f0d59e1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fxm7k\" (UID: \"6efbc03a-f9b7-463d-969e-7c420f0d59e1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k"
Nov 03 19:46:14.560500 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:14.560432    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6efbc03a-f9b7-463d-969e-7c420f0d59e1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fxm7k\" (UID: \"6efbc03a-f9b7-463d-969e-7c420f0d59e1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k"
Nov 03 19:46:14.560857 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:14.560833    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6efbc03a-f9b7-463d-969e-7c420f0d59e1-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fxm7k\" (UID: \"6efbc03a-f9b7-463d-969e-7c420f0d59e1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k"
Nov 03 19:46:14.560857 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:14.560854    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6efbc03a-f9b7-463d-969e-7c420f0d59e1-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fxm7k\" (UID: \"6efbc03a-f9b7-463d-969e-7c420f0d59e1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k"
Nov 03 19:46:14.561032 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:14.561014    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6efbc03a-f9b7-463d-969e-7c420f0d59e1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fxm7k\" (UID: \"6efbc03a-f9b7-463d-969e-7c420f0d59e1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k"
Nov 03 19:46:14.562861 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:14.562838    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6efbc03a-f9b7-463d-969e-7c420f0d59e1-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fxm7k\" (UID: \"6efbc03a-f9b7-463d-969e-7c420f0d59e1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k"
Nov 03 19:46:14.563084 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:14.563065    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6efbc03a-f9b7-463d-969e-7c420f0d59e1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fxm7k\" (UID: \"6efbc03a-f9b7-463d-969e-7c420f0d59e1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k"
Nov 03 19:46:14.575614 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:14.575587    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn4nc\" (UniqueName: \"kubernetes.io/projected/6efbc03a-f9b7-463d-969e-7c420f0d59e1-kube-api-access-kn4nc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-fxm7k\" (UID: \"6efbc03a-f9b7-463d-969e-7c420f0d59e1\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k"
Nov 03 19:46:14.746684 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:14.746588    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k"
Nov 03 19:46:15.032025 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:15.022632    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k"]
Nov 03 19:46:15.280958 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:15.280873    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:46:15.518472 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:15.518435    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm"
Nov 03 19:46:15.526899 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:15.526862    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k" event={"ID":"6efbc03a-f9b7-463d-969e-7c420f0d59e1","Type":"ContainerStarted","Data":"dce0dc3b5cfa3b05475ec934cdfdfb248eb87b241cd8e95ce4839bdba9848e1a"}
Nov 03 19:46:15.527057 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:15.526919    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k" event={"ID":"6efbc03a-f9b7-463d-969e-7c420f0d59e1","Type":"ContainerStarted","Data":"2b6925c23bb0f3467cd1f2ca9dc1f1d034170c590858131c71649ef202f061b3"}
Nov 03 19:46:15.547916 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:15.547824    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k" podStartSLOduration=1.213235842 podStartE2EDuration="1.547809546s" totalImagesPullingTime="334.573704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:15.546604262 +0000 UTC m=+524.318897281" watchObservedRunningTime="2026-11-03 19:46:15.547809546 +0000 UTC m=+524.320102564"
Nov 03 19:46:15.747660 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:15.747619    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k"
Nov 03 19:46:19.315026 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:19.314991    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg"
Nov 03 19:46:19.328235 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:19.328177    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg"
Nov 03 19:46:19.401221 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:19.401182    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:46:19.557345 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:19.557313    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg"
Nov 03 19:46:19.904966 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:19.904937    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks"]
Nov 03 19:46:19.912283 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:19.912254    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks"
Nov 03 19:46:19.917957 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:19.917928    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 03 19:46:19.921114 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:19.921080    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks"]
Nov 03 19:46:20.012402 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:20.012371    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c1de34b-e0ff-4411-b4c1-1318253b182a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-krbks\" (UID: \"8c1de34b-e0ff-4411-b4c1-1318253b182a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks"
Nov 03 19:46:20.012598 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:20.012409    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7slmv\" (UniqueName: \"kubernetes.io/projected/8c1de34b-e0ff-4411-b4c1-1318253b182a-kube-api-access-7slmv\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-krbks\" (UID: \"8c1de34b-e0ff-4411-b4c1-1318253b182a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks"
Nov 03 19:46:20.012598 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:20.012431    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c1de34b-e0ff-4411-b4c1-1318253b182a-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-krbks\" (UID: \"8c1de34b-e0ff-4411-b4c1-1318253b182a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks"
Nov 03 19:46:20.012598 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:20.012504    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c1de34b-e0ff-4411-b4c1-1318253b182a-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-krbks\" (UID: \"8c1de34b-e0ff-4411-b4c1-1318253b182a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks"
Nov 03 19:46:20.012598 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:20.012534    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c1de34b-e0ff-4411-b4c1-1318253b182a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-krbks\" (UID: \"8c1de34b-e0ff-4411-b4c1-1318253b182a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks"
Nov 03 19:46:20.012752 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:20.012676    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c1de34b-e0ff-4411-b4c1-1318253b182a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-krbks\" (UID: \"8c1de34b-e0ff-4411-b4c1-1318253b182a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks"
Nov 03 19:46:20.113745 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:20.113710    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c1de34b-e0ff-4411-b4c1-1318253b182a-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-krbks\" (UID: \"8c1de34b-e0ff-4411-b4c1-1318253b182a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks"
Nov 03 19:46:20.113745 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:20.113748    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c1de34b-e0ff-4411-b4c1-1318253b182a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-krbks\" (UID: \"8c1de34b-e0ff-4411-b4c1-1318253b182a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks"
Nov 03 19:46:20.113975 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:20.113786    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c1de34b-e0ff-4411-b4c1-1318253b182a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-krbks\" (UID: \"8c1de34b-e0ff-4411-b4c1-1318253b182a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks"
Nov 03 19:46:20.113975 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:20.113836    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c1de34b-e0ff-4411-b4c1-1318253b182a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-krbks\" (UID: \"8c1de34b-e0ff-4411-b4c1-1318253b182a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks"
Nov 03 19:46:20.113975 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:20.113862    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7slmv\" (UniqueName: \"kubernetes.io/projected/8c1de34b-e0ff-4411-b4c1-1318253b182a-kube-api-access-7slmv\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-krbks\" (UID: \"8c1de34b-e0ff-4411-b4c1-1318253b182a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks"
Nov 03 19:46:20.114128 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:20.113986    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c1de34b-e0ff-4411-b4c1-1318253b182a-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-krbks\" (UID: \"8c1de34b-e0ff-4411-b4c1-1318253b182a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks"
Nov 03 19:46:20.114227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:20.114184    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c1de34b-e0ff-4411-b4c1-1318253b182a-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-krbks\" (UID: \"8c1de34b-e0ff-4411-b4c1-1318253b182a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks"
Nov 03 19:46:20.114306 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:20.114259    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c1de34b-e0ff-4411-b4c1-1318253b182a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-krbks\" (UID: \"8c1de34b-e0ff-4411-b4c1-1318253b182a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks"
Nov 03 19:46:20.114349 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:20.114300    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c1de34b-e0ff-4411-b4c1-1318253b182a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-krbks\" (UID: \"8c1de34b-e0ff-4411-b4c1-1318253b182a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks"
Nov 03 19:46:20.116292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:20.116263    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c1de34b-e0ff-4411-b4c1-1318253b182a-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-krbks\" (UID: \"8c1de34b-e0ff-4411-b4c1-1318253b182a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks"
Nov 03 19:46:20.116432 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:20.116416    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c1de34b-e0ff-4411-b4c1-1318253b182a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-krbks\" (UID: \"8c1de34b-e0ff-4411-b4c1-1318253b182a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks"
Nov 03 19:46:20.127044 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:20.127015    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7slmv\" (UniqueName: \"kubernetes.io/projected/8c1de34b-e0ff-4411-b4c1-1318253b182a-kube-api-access-7slmv\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-krbks\" (UID: \"8c1de34b-e0ff-4411-b4c1-1318253b182a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks"
Nov 03 19:46:20.226851 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:20.226764    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks"
Nov 03 19:46:20.511817 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:20.511736    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks"]
Nov 03 19:46:20.515594 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:46:20.515559    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1de34b_e0ff_4411_b4c1_1318253b182a.slice/crio-0784c9e77894f2b000bfea41080b05177cb2d04184e51dff659e2b41982a038d WatchSource:0}: Error finding container 0784c9e77894f2b000bfea41080b05177cb2d04184e51dff659e2b41982a038d: Status 404 returned error can't find the container with id 0784c9e77894f2b000bfea41080b05177cb2d04184e51dff659e2b41982a038d
Nov 03 19:46:20.517388 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:20.517368    5591 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:46:20.551853 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:20.551818    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks" event={"ID":"8c1de34b-e0ff-4411-b4c1-1318253b182a","Type":"ContainerStarted","Data":"0784c9e77894f2b000bfea41080b05177cb2d04184e51dff659e2b41982a038d"}
Nov 03 19:46:21.557655 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:21.557619    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks" event={"ID":"8c1de34b-e0ff-4411-b4c1-1318253b182a","Type":"ContainerStarted","Data":"02c97ac256b5675d8f3e2707081dbebfdf93554805946f2615d600fe7157dd3e"}
Nov 03 19:46:21.577723 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:21.577667    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks" podStartSLOduration=2.259305458 podStartE2EDuration="2.577653888s" totalImagesPullingTime="318.34843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:21.576644547 +0000 UTC m=+530.348937576" watchObservedRunningTime="2026-11-03 19:46:21.577653888 +0000 UTC m=+530.349946900"
Nov 03 19:46:22.227437 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:22.227406    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks"
Nov 03 19:46:23.518402 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:23.518366    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm"
Nov 03 19:46:23.530816 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:23.530788    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm"
Nov 03 19:46:23.578870 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:23.578849    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm"
Nov 03 19:46:24.747285 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:24.747247    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k"
Nov 03 19:46:24.759772 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:24.759748    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k"
Nov 03 19:46:25.324076 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:25.324038    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:46:25.587233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:25.587128    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k"
Nov 03 19:46:28.722265 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:46:28.722228    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:46:30.227723 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:30.227686    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks"
Nov 03 19:46:30.240162 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:30.240132    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks"
Nov 03 19:46:30.608930 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:30.608903    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-krbks"
Nov 03 19:46:39.001874 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.001832    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"]
Nov 03 19:46:39.006906 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.006837    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.023816 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.023780    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"]
Nov 03 19:46:39.090288 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.090245    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e14ca8f3-c80b-4c0b-9564-ef9734b86de0-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-mrwmt\" (UID: \"e14ca8f3-c80b-4c0b-9564-ef9734b86de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.090459 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.090307    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14ca8f3-c80b-4c0b-9564-ef9734b86de0-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-mrwmt\" (UID: \"e14ca8f3-c80b-4c0b-9564-ef9734b86de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.090459 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.090342    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e14ca8f3-c80b-4c0b-9564-ef9734b86de0-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-mrwmt\" (UID: \"e14ca8f3-c80b-4c0b-9564-ef9734b86de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.090459 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.090383    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e14ca8f3-c80b-4c0b-9564-ef9734b86de0-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-mrwmt\" (UID: \"e14ca8f3-c80b-4c0b-9564-ef9734b86de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.090581 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.090500    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e14ca8f3-c80b-4c0b-9564-ef9734b86de0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-mrwmt\" (UID: \"e14ca8f3-c80b-4c0b-9564-ef9734b86de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.090581 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.090545    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e14ca8f3-c80b-4c0b-9564-ef9734b86de0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-mrwmt\" (UID: \"e14ca8f3-c80b-4c0b-9564-ef9734b86de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.090659 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.090626    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e14ca8f3-c80b-4c0b-9564-ef9734b86de0-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-mrwmt\" (UID: \"e14ca8f3-c80b-4c0b-9564-ef9734b86de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.090659 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.090643    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qr4j\" (UniqueName: \"kubernetes.io/projected/e14ca8f3-c80b-4c0b-9564-ef9734b86de0-kube-api-access-8qr4j\") pod \"maas-default-gateway-openshift-default-855f9574cf-mrwmt\" (UID: \"e14ca8f3-c80b-4c0b-9564-ef9734b86de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.090735 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.090666    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14ca8f3-c80b-4c0b-9564-ef9734b86de0-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-mrwmt\" (UID: \"e14ca8f3-c80b-4c0b-9564-ef9734b86de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.103722 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.103701    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-65hnm\""
Nov 03 19:46:39.191369 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.191338    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e14ca8f3-c80b-4c0b-9564-ef9734b86de0-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-mrwmt\" (UID: \"e14ca8f3-c80b-4c0b-9564-ef9734b86de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.191500 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.191382    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e14ca8f3-c80b-4c0b-9564-ef9734b86de0-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-mrwmt\" (UID: \"e14ca8f3-c80b-4c0b-9564-ef9734b86de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.191500 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.191413    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e14ca8f3-c80b-4c0b-9564-ef9734b86de0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-mrwmt\" (UID: \"e14ca8f3-c80b-4c0b-9564-ef9734b86de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.191500 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.191433    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e14ca8f3-c80b-4c0b-9564-ef9734b86de0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-mrwmt\" (UID: \"e14ca8f3-c80b-4c0b-9564-ef9734b86de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.191500 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.191462    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e14ca8f3-c80b-4c0b-9564-ef9734b86de0-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-mrwmt\" (UID: \"e14ca8f3-c80b-4c0b-9564-ef9734b86de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.191500 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.191485    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qr4j\" (UniqueName: \"kubernetes.io/projected/e14ca8f3-c80b-4c0b-9564-ef9734b86de0-kube-api-access-8qr4j\") pod \"maas-default-gateway-openshift-default-855f9574cf-mrwmt\" (UID: \"e14ca8f3-c80b-4c0b-9564-ef9734b86de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.191726 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.191509    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14ca8f3-c80b-4c0b-9564-ef9734b86de0-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-mrwmt\" (UID: \"e14ca8f3-c80b-4c0b-9564-ef9734b86de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.191726 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.191587    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e14ca8f3-c80b-4c0b-9564-ef9734b86de0-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-mrwmt\" (UID: \"e14ca8f3-c80b-4c0b-9564-ef9734b86de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.191726 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.191623    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14ca8f3-c80b-4c0b-9564-ef9734b86de0-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-mrwmt\" (UID: \"e14ca8f3-c80b-4c0b-9564-ef9734b86de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.191931 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.191897    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e14ca8f3-c80b-4c0b-9564-ef9734b86de0-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-mrwmt\" (UID: \"e14ca8f3-c80b-4c0b-9564-ef9734b86de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.192072 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.192005    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14ca8f3-c80b-4c0b-9564-ef9734b86de0-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-mrwmt\" (UID: \"e14ca8f3-c80b-4c0b-9564-ef9734b86de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.192129 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.192107    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e14ca8f3-c80b-4c0b-9564-ef9734b86de0-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-mrwmt\" (UID: \"e14ca8f3-c80b-4c0b-9564-ef9734b86de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.192250 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.192197    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e14ca8f3-c80b-4c0b-9564-ef9734b86de0-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-mrwmt\" (UID: \"e14ca8f3-c80b-4c0b-9564-ef9734b86de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.192342 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.192309    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e14ca8f3-c80b-4c0b-9564-ef9734b86de0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-mrwmt\" (UID: \"e14ca8f3-c80b-4c0b-9564-ef9734b86de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.193933 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.193913    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e14ca8f3-c80b-4c0b-9564-ef9734b86de0-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-mrwmt\" (UID: \"e14ca8f3-c80b-4c0b-9564-ef9734b86de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.194182 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.194162    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e14ca8f3-c80b-4c0b-9564-ef9734b86de0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-mrwmt\" (UID: \"e14ca8f3-c80b-4c0b-9564-ef9734b86de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.203334 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.203310    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e14ca8f3-c80b-4c0b-9564-ef9734b86de0-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-mrwmt\" (UID: \"e14ca8f3-c80b-4c0b-9564-ef9734b86de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.203439 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.203422    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qr4j\" (UniqueName: \"kubernetes.io/projected/e14ca8f3-c80b-4c0b-9564-ef9734b86de0-kube-api-access-8qr4j\") pod \"maas-default-gateway-openshift-default-855f9574cf-mrwmt\" (UID: \"e14ca8f3-c80b-4c0b-9564-ef9734b86de0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.321407 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.321379    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:39.612693 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.612667    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"]
Nov 03 19:46:39.615178 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:46:39.615147    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode14ca8f3_c80b_4c0b_9564_ef9734b86de0.slice/crio-5879d072a3a4040a6c0c572895baf060259fb0f0a02e582ab1c2760172bac915 WatchSource:0}: Error finding container 5879d072a3a4040a6c0c572895baf060259fb0f0a02e582ab1c2760172bac915: Status 404 returned error can't find the container with id 5879d072a3a4040a6c0c572895baf060259fb0f0a02e582ab1c2760172bac915
Nov 03 19:46:39.640858 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:39.640825    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt" event={"ID":"e14ca8f3-c80b-4c0b-9564-ef9734b86de0","Type":"ContainerStarted","Data":"5879d072a3a4040a6c0c572895baf060259fb0f0a02e582ab1c2760172bac915"}
Nov 03 19:46:47.276409 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:47.276373    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:46:47.276721 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:47.276449    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:46:47.276721 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:47.276484    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:46:47.702077 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:47.701970    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt" event={"ID":"e14ca8f3-c80b-4c0b-9564-ef9734b86de0","Type":"ContainerStarted","Data":"3b40f2334f0469b197e8957a0d5a3cc88ef197e3e38fea9534d9fa9015967f31"}
Nov 03 19:46:47.727489 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:47.727430    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt" podStartSLOduration=2.069160904 podStartE2EDuration="9.727414318s" totalImagesPullingTime="7.658253414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:47.726298228 +0000 UTC m=+556.498591296" watchObservedRunningTime="2026-11-03 19:46:47.727414318 +0000 UTC m=+556.499707399"
Nov 03 19:46:48.322097 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:48.322057    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:48.323600 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:48.323573    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-mrwmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 19:46:48.323740 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:48.323622    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt" podUID="e14ca8f3-c80b-4c0b-9564-ef9734b86de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 19:46:49.322432 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:49.322394    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-mrwmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 19:46:49.322831 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:49.322452    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt" podUID="e14ca8f3-c80b-4c0b-9564-ef9734b86de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 19:46:50.322063 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:50.322022    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-mrwmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 19:46:50.322286 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:50.322083    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt" podUID="e14ca8f3-c80b-4c0b-9564-ef9734b86de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 19:46:51.322504 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:51.322467    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-mrwmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 19:46:51.322902 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:51.322520    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt" podUID="e14ca8f3-c80b-4c0b-9564-ef9734b86de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 19:46:52.322427 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:52.322386    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-mrwmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 19:46:52.322598 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:52.322472    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt" podUID="e14ca8f3-c80b-4c0b-9564-ef9734b86de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 19:46:53.322640 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:53.322607    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-mrwmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 19:46:53.323047 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:53.322661    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt" podUID="e14ca8f3-c80b-4c0b-9564-ef9734b86de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 19:46:54.033093 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.033050    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"]
Nov 03 19:46:54.037021 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.037000    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.055402 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.055368    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"]
Nov 03 19:46:54.056234 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.056177    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a0f28bc-d05f-45bb-8431-3aaf73529510-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-9cvzn\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.056345 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.056295    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-9cvzn\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.056413 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.056345    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-9cvzn\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.056413 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.056395    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-9cvzn\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.056522 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.056439    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-9cvzn\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.056647 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.056610    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-9cvzn\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.056727 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.056695    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slwhg\" (UniqueName: \"kubernetes.io/projected/3a0f28bc-d05f-45bb-8431-3aaf73529510-kube-api-access-slwhg\") pod \"maas-default-gateway-openshift-default-855f9574cf-9cvzn\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.056863 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.056841    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-9cvzn\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.056922 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.056895    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-9cvzn\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.157609 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.157570    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-9cvzn\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.157609 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.157615    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-9cvzn\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.157866 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.157639    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-9cvzn\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.157866 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.157799    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-9cvzn\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.157866 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.157849    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-9cvzn\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.158026 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.157899    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slwhg\" (UniqueName: \"kubernetes.io/projected/3a0f28bc-d05f-45bb-8431-3aaf73529510-kube-api-access-slwhg\") pod \"maas-default-gateway-openshift-default-855f9574cf-9cvzn\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.158026 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.157932    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-9cvzn\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.158026 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.157963    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-9cvzn\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.158173 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.158042    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a0f28bc-d05f-45bb-8431-3aaf73529510-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-9cvzn\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.158173 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.158131    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-9cvzn\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.158316 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.158183    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-9cvzn\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.158772 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.158748    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-9cvzn\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.158995 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.158973    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a0f28bc-d05f-45bb-8431-3aaf73529510-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-9cvzn\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.159369 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.159348    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-9cvzn\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.160149 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.160127    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-9cvzn\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.161078 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.161059    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-9cvzn\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.170885 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.170862    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-9cvzn\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.170989 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.170888    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slwhg\" (UniqueName: \"kubernetes.io/projected/3a0f28bc-d05f-45bb-8431-3aaf73529510-kube-api-access-slwhg\") pod \"maas-default-gateway-openshift-default-855f9574cf-9cvzn\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.322403 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.322309    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:54.322601 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.322579    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-mrwmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 19:46:54.322667 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.322629    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt" podUID="e14ca8f3-c80b-4c0b-9564-ef9734b86de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 19:46:54.363950 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.363915    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:54.653396 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.653047    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"]
Nov 03 19:46:54.655633 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:46:54.655606    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0f28bc_d05f_45bb_8431_3aaf73529510.slice/crio-85cb93708fd1bb293bc411ecc1ea05defe98681b457b6e0e62104934ef4bc6a7 WatchSource:0}: Error finding container 85cb93708fd1bb293bc411ecc1ea05defe98681b457b6e0e62104934ef4bc6a7: Status 404 returned error can't find the container with id 85cb93708fd1bb293bc411ecc1ea05defe98681b457b6e0e62104934ef4bc6a7
Nov 03 19:46:54.658743 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.658705    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:46:54.658824 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.658794    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:46:54.658865 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.658834    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:46:54.736485 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.736447    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn" event={"ID":"3a0f28bc-d05f-45bb-8431-3aaf73529510","Type":"ContainerStarted","Data":"00bc6fbf7920866477e22c63b8322d9015bf9059e8424f6465b6eb5fe6a77e8e"}
Nov 03 19:46:54.736648 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.736527    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn" event={"ID":"3a0f28bc-d05f-45bb-8431-3aaf73529510","Type":"ContainerStarted","Data":"85cb93708fd1bb293bc411ecc1ea05defe98681b457b6e0e62104934ef4bc6a7"}
Nov 03 19:46:54.760564 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:54.760506    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn" podStartSLOduration=0.76049148 podStartE2EDuration="760.49148ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:54.759701804 +0000 UTC m=+563.531994838" watchObservedRunningTime="2026-11-03 19:46:54.76049148 +0000 UTC m=+563.532784497"
Nov 03 19:46:55.364997 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:55.364971    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:46:55.366499 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:55.366477    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-9cvzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 19:46:55.366618 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:55.366534    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn" podUID="3a0f28bc-d05f-45bb-8431-3aaf73529510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 19:46:55.733554 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:55.733468    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:55.734731 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:55.734709    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt"
Nov 03 19:46:56.364888 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:56.364850    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-9cvzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 19:46:56.365070 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:56.364921    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn" podUID="3a0f28bc-d05f-45bb-8431-3aaf73529510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 19:46:57.364723 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:57.364685    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-9cvzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 19:46:57.364919 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:57.364747    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn" podUID="3a0f28bc-d05f-45bb-8431-3aaf73529510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 19:46:58.365337 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:58.365304    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-9cvzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 19:46:58.365740 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:58.365359    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn" podUID="3a0f28bc-d05f-45bb-8431-3aaf73529510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 19:46:58.725777 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:46:58.725695    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:46:59.364899 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:59.364865    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-9cvzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 19:46:59.365098 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:46:59.364923    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn" podUID="3a0f28bc-d05f-45bb-8431-3aaf73529510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 19:47:00.365071 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:00.365031    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-9cvzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 19:47:00.365500 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:00.365089    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn" podUID="3a0f28bc-d05f-45bb-8431-3aaf73529510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 19:47:01.364850 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:01.364811    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-9cvzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 03 19:47:01.365029 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:01.364884    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn" podUID="3a0f28bc-d05f-45bb-8431-3aaf73529510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 03 19:47:02.996057 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:02.996028    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:47:02.996564 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:02.996463    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:47:02.997458 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:02.997426    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 19:47:04.875485 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:04.875452    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"]
Nov 03 19:47:04.879907 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:04.879883    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:04.889585 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:04.889561    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c20ed791-openshift-default-dockercfg-nfq98\""
Nov 03 19:47:04.898721 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:04.898693    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"]
Nov 03 19:47:04.989323 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:04.989280    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-token\") pod \"e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") " pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:04.989484 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:04.989343    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-workload-socket\") pod \"e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") " pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:04.989542 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:04.989519    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-podinfo\") pod \"e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") " pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:04.989582 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:04.989562    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-envoy\") pod \"e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") " pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:04.989637 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:04.989620    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-data\") pod \"e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") " pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:04.989669 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:04.989653    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwn9t\" (UniqueName: \"kubernetes.io/projected/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-kube-api-access-jwn9t\") pod \"e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") " pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:04.989800 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:04.989787    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-credential-socket\") pod \"e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") " pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:04.989833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:04.989813    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-workload-certs\") pod \"e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") " pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:04.989833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:04.989829    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istiod-ca-cert\") pod \"e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") " pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:05.093311 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:05.093148    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-credential-socket\") pod \"e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") " pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:05.093311 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:05.093221    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-workload-certs\") pod \"e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") " pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:05.093311 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:05.093248    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istiod-ca-cert\") pod \"e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") " pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:05.093311 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:05.093285    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-token\") pod \"e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") " pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:05.093311 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:05.093326    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-workload-socket\") pod \"e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") " pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:05.093778 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:05.093381    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-podinfo\") pod \"e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") " pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:05.093778 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:05.093407    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-envoy\") pod \"e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") " pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:05.093778 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:05.093429    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-data\") pod \"e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") " pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:05.093778 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:05.093452    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwn9t\" (UniqueName: \"kubernetes.io/projected/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-kube-api-access-jwn9t\") pod \"e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") " pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:05.094168 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:05.094139    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-credential-socket\") pod \"e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") " pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:05.094446 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:05.094422    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-data\") pod \"e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") " pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:05.094602 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:05.094584    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-workload-socket\") pod \"e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") " pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:05.094724 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:05.094699    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istiod-ca-cert\") pod \"e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") " pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:05.094828 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:05.094807    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-workload-certs\") pod \"e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") " pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:05.096955 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:05.096928    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-podinfo\") pod \"e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") " pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:05.097535 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:05.097505    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-envoy\") pod \"e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") " pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:05.107946 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:05.107921    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-token\") pod \"e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") " pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:05.108451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:05.108429    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwn9t\" (UniqueName: \"kubernetes.io/projected/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-kube-api-access-jwn9t\") pod \"e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") " pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:05.195378 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:05.194694    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:05.533232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:05.533195    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"]
Nov 03 19:47:05.535168 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:47:05.535141    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b7ef8f_17e3_4fe9_a98d_cbd45650f626.slice/crio-7e020a100ebd15cb30c8222685dd7f07b9fe8815c46cfe94d48b570e58284dba WatchSource:0}: Error finding container 7e020a100ebd15cb30c8222685dd7f07b9fe8815c46cfe94d48b570e58284dba: Status 404 returned error can't find the container with id 7e020a100ebd15cb30c8222685dd7f07b9fe8815c46cfe94d48b570e58284dba
Nov 03 19:47:05.537425 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:05.537393    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:47:05.537540 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:05.537453    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:47:05.537540 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:05.537483    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:47:05.790766 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:05.790684    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f" event={"ID":"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626","Type":"ContainerStarted","Data":"07bd397d52b25687427b76006cc0518d1c65812a4f209c3692e3491100e3b05f"}
Nov 03 19:47:05.790766 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:05.790751    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f" event={"ID":"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626","Type":"ContainerStarted","Data":"7e020a100ebd15cb30c8222685dd7f07b9fe8815c46cfe94d48b570e58284dba"}
Nov 03 19:47:05.820320 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:05.820262    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f" podStartSLOduration=1.820242868 podStartE2EDuration="1.820242868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:05.818465666 +0000 UTC m=+574.590758685" watchObservedRunningTime="2026-11-03 19:47:05.820242868 +0000 UTC m=+574.592535887"
Nov 03 19:47:06.195193 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.195110    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:06.200376 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.200347    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:06.599375 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.599341    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"]
Nov 03 19:47:06.604228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.604191    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.609018 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.608997    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3f7e10b8-openshift-default-dockercfg-d25kx\""
Nov 03 19:47:06.618083 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.618053    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"]
Nov 03 19:47:06.709834 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.709792    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-workload-certs\") pod \"e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") " pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.710012 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.709847    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27e0a187-19d5-45cb-bdd6-711f30368cca-istiod-ca-cert\") pod \"e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") " pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.710012 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.709965    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-token\") pod \"e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") " pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.710148 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.710030    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-workload-socket\") pod \"e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") " pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.710148 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.710076    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm9hs\" (UniqueName: \"kubernetes.io/projected/27e0a187-19d5-45cb-bdd6-711f30368cca-kube-api-access-bm9hs\") pod \"e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") " pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.710288 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.710159    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-envoy\") pod \"e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") " pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.710288 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.710190    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-credential-socket\") pod \"e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") " pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.710396 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.710312    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-podinfo\") pod \"e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") " pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.710396 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.710369    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-data\") pod \"e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") " pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.802956 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.802928    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:06.804093 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.804069    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:06.811665 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.811479    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-workload-socket\") pod \"e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") " pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.811665 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.811535    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm9hs\" (UniqueName: \"kubernetes.io/projected/27e0a187-19d5-45cb-bdd6-711f30368cca-kube-api-access-bm9hs\") pod \"e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") " pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.811665 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.811591    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-envoy\") pod \"e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") " pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.811665 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.811624    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-credential-socket\") pod \"e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") " pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.811665 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.811660    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-podinfo\") pod \"e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") " pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.813323 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.811711    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-data\") pod \"e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") " pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.813323 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.811763    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-workload-certs\") pod \"e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") " pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.813323 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.811819    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27e0a187-19d5-45cb-bdd6-711f30368cca-istiod-ca-cert\") pod \"e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") " pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.813323 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.811846    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-token\") pod \"e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") " pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.813323 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.812048    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-workload-socket\") pod \"e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") " pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.813323 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.812463    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-data\") pod \"e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") " pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.813323 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.812856    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-workload-certs\") pod \"e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") " pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.813323 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.813051    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-credential-socket\") pod \"e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") " pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.813323 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.813125    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27e0a187-19d5-45cb-bdd6-711f30368cca-istiod-ca-cert\") pod \"e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") " pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.823255 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.816071    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-podinfo\") pod \"e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") " pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.831176 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.831139    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-envoy\") pod \"e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") " pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.834327 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.832694    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm9hs\" (UniqueName: \"kubernetes.io/projected/27e0a187-19d5-45cb-bdd6-711f30368cca-kube-api-access-bm9hs\") pod \"e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") " pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.834327 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.833657    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-token\") pod \"e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") " pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:06.931729 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:06.931647    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:07.215095 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:07.215001    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"]
Nov 03 19:47:07.217758 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:47:07.217719    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e0a187_19d5_45cb_bdd6_711f30368cca.slice/crio-45304c163d644d5fcdf4a04d2becc0c1c9f59f9ea088e2ca0ad06eb16a174d36 WatchSource:0}: Error finding container 45304c163d644d5fcdf4a04d2becc0c1c9f59f9ea088e2ca0ad06eb16a174d36: Status 404 returned error can't find the container with id 45304c163d644d5fcdf4a04d2becc0c1c9f59f9ea088e2ca0ad06eb16a174d36
Nov 03 19:47:07.220118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:07.220083    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:47:07.220259 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:07.220154    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:47:07.220259 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:07.220186    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:47:07.808277 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:07.808239    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr" event={"ID":"27e0a187-19d5-45cb-bdd6-711f30368cca","Type":"ContainerStarted","Data":"f21e61b950bb814337c21e1a66b772f692528552543377abd1677bf686013934"}
Nov 03 19:47:07.808474 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:07.808298    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr" event={"ID":"27e0a187-19d5-45cb-bdd6-711f30368cca","Type":"ContainerStarted","Data":"45304c163d644d5fcdf4a04d2becc0c1c9f59f9ea088e2ca0ad06eb16a174d36"}
Nov 03 19:47:07.833849 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:07.833799    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr" podStartSLOduration=1.833786208 podStartE2EDuration="1.833786208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:07.832268739 +0000 UTC m=+576.604561760" watchObservedRunningTime="2026-11-03 19:47:07.833786208 +0000 UTC m=+576.606079226"
Nov 03 19:47:07.932434 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:07.932408    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:08.584369 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:08.584337    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d"]
Nov 03 19:47:08.588710 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:08.588688    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d"
Nov 03 19:47:08.597866 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:08.597825    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c20ed791\""
Nov 03 19:47:08.598031 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:08.597921    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c20ed791-dockercfg-vzz24\""
Nov 03 19:47:08.600053 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:08.600031    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d"]
Nov 03 19:47:08.618534 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:08.618507    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7"]
Nov 03 19:47:08.622177 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:08.622156    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7"
Nov 03 19:47:08.633362 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:08.633318    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7"]
Nov 03 19:47:08.733866 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:08.733833    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct7r4\" (UniqueName: \"kubernetes.io/projected/5b020e42-25c7-41b6-93fa-ebc2749579a4-kube-api-access-ct7r4\") pod \"payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7\" (UID: \"5b020e42-25c7-41b6-93fa-ebc2749579a4\") " pod="openshift-ingress/payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7"
Nov 03 19:47:08.734030 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:08.733877    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b020e42-25c7-41b6-93fa-ebc2749579a4-plugins-config-volume\") pod \"payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7\" (UID: \"5b020e42-25c7-41b6-93fa-ebc2749579a4\") " pod="openshift-ingress/payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7"
Nov 03 19:47:08.734030 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:08.733955    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmtrd\" (UniqueName: \"kubernetes.io/projected/41174c3b-6b28-4a8c-82e8-acbf40ba454f-kube-api-access-jmtrd\") pod \"payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d\" (UID: \"41174c3b-6b28-4a8c-82e8-acbf40ba454f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d"
Nov 03 19:47:08.734163 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:08.734141    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41174c3b-6b28-4a8c-82e8-acbf40ba454f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d\" (UID: \"41174c3b-6b28-4a8c-82e8-acbf40ba454f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d"
Nov 03 19:47:08.835268 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:08.835150    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41174c3b-6b28-4a8c-82e8-acbf40ba454f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d\" (UID: \"41174c3b-6b28-4a8c-82e8-acbf40ba454f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d"
Nov 03 19:47:08.835268 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:08.835188    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct7r4\" (UniqueName: \"kubernetes.io/projected/5b020e42-25c7-41b6-93fa-ebc2749579a4-kube-api-access-ct7r4\") pod \"payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7\" (UID: \"5b020e42-25c7-41b6-93fa-ebc2749579a4\") " pod="openshift-ingress/payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7"
Nov 03 19:47:08.835268 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:08.835240    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b020e42-25c7-41b6-93fa-ebc2749579a4-plugins-config-volume\") pod \"payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7\" (UID: \"5b020e42-25c7-41b6-93fa-ebc2749579a4\") " pod="openshift-ingress/payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7"
Nov 03 19:47:08.835268 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:08.835265    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmtrd\" (UniqueName: \"kubernetes.io/projected/41174c3b-6b28-4a8c-82e8-acbf40ba454f-kube-api-access-jmtrd\") pod \"payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d\" (UID: \"41174c3b-6b28-4a8c-82e8-acbf40ba454f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d"
Nov 03 19:47:08.836115 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:08.836094    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41174c3b-6b28-4a8c-82e8-acbf40ba454f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d\" (UID: \"41174c3b-6b28-4a8c-82e8-acbf40ba454f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d"
Nov 03 19:47:08.836270 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:08.836137    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b020e42-25c7-41b6-93fa-ebc2749579a4-plugins-config-volume\") pod \"payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7\" (UID: \"5b020e42-25c7-41b6-93fa-ebc2749579a4\") " pod="openshift-ingress/payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7"
Nov 03 19:47:08.848607 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:08.848571    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmtrd\" (UniqueName: \"kubernetes.io/projected/41174c3b-6b28-4a8c-82e8-acbf40ba454f-kube-api-access-jmtrd\") pod \"payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d\" (UID: \"41174c3b-6b28-4a8c-82e8-acbf40ba454f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d"
Nov 03 19:47:08.848775 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:08.848719    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct7r4\" (UniqueName: \"kubernetes.io/projected/5b020e42-25c7-41b6-93fa-ebc2749579a4-kube-api-access-ct7r4\") pod \"payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7\" (UID: \"5b020e42-25c7-41b6-93fa-ebc2749579a4\") " pod="openshift-ingress/payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7"
Nov 03 19:47:08.904485 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:08.904454    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d"
Nov 03 19:47:08.936601 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:08.936575    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7"
Nov 03 19:47:08.939719 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:08.939692    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:09.193332 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.193303    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d"]
Nov 03 19:47:09.196134 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:47:09.196102    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41174c3b_6b28_4a8c_82e8_acbf40ba454f.slice/crio-d6c5a8e171a37903112deb01de937685f48e39e6954d298fba329bc88125c8cc WatchSource:0}: Error finding container d6c5a8e171a37903112deb01de937685f48e39e6954d298fba329bc88125c8cc: Status 404 returned error can't find the container with id d6c5a8e171a37903112deb01de937685f48e39e6954d298fba329bc88125c8cc
Nov 03 19:47:09.229975 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.229950    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7"]
Nov 03 19:47:09.294019 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.293991    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l"]
Nov 03 19:47:09.298834 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.298801    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l"
Nov 03 19:47:09.304031 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.304003    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2"]
Nov 03 19:47:09.308644 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.308618    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-745a88f9\""
Nov 03 19:47:09.308800 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.308700    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-745a88f9-dockercfg-4kwk8\""
Nov 03 19:47:09.310275 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.310255    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l"]
Nov 03 19:47:09.310371 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.310355    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2"
Nov 03 19:47:09.330575 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.330547    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2"]
Nov 03 19:47:09.451033 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.450933    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d9c6d40-8002-4c1d-a314-b083c06c28c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2\" (UID: \"7d9c6d40-8002-4c1d-a314-b083c06c28c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2"
Nov 03 19:47:09.451767 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.451340    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wq74\" (UniqueName: \"kubernetes.io/projected/feaac383-b97d-42ad-b021-2e24d707739b-kube-api-access-6wq74\") pod \"payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l\" (UID: \"feaac383-b97d-42ad-b021-2e24d707739b\") " pod="openshift-ingress/payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l"
Nov 03 19:47:09.451952 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.451928    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sppsg\" (UniqueName: \"kubernetes.io/projected/7d9c6d40-8002-4c1d-a314-b083c06c28c1-kube-api-access-sppsg\") pod \"payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2\" (UID: \"7d9c6d40-8002-4c1d-a314-b083c06c28c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2"
Nov 03 19:47:09.452044 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.452029    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feaac383-b97d-42ad-b021-2e24d707739b-plugins-config-volume\") pod \"payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l\" (UID: \"feaac383-b97d-42ad-b021-2e24d707739b\") " pod="openshift-ingress/payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l"
Nov 03 19:47:09.470673 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.470623    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb"]
Nov 03 19:47:09.475447 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.475419    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb"
Nov 03 19:47:09.484460 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.484437    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 19:47:09.487672 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.487618    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb"]
Nov 03 19:47:09.487785 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.487754    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-942q2\""
Nov 03 19:47:09.488033 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.488009    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:47:09.488120 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.488089    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:47:09.488183 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.488157    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c20ed791-serving-cert\""
Nov 03 19:47:09.552879 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.552845    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wq74\" (UniqueName: \"kubernetes.io/projected/feaac383-b97d-42ad-b021-2e24d707739b-kube-api-access-6wq74\") pod \"payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l\" (UID: \"feaac383-b97d-42ad-b021-2e24d707739b\") " pod="openshift-ingress/payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l"
Nov 03 19:47:09.552879 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.552882    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssmlc\" (UniqueName: \"kubernetes.io/projected/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-kube-api-access-ssmlc\") pod \"maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb\" (UID: \"b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb"
Nov 03 19:47:09.553098 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.552906    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sppsg\" (UniqueName: \"kubernetes.io/projected/7d9c6d40-8002-4c1d-a314-b083c06c28c1-kube-api-access-sppsg\") pod \"payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2\" (UID: \"7d9c6d40-8002-4c1d-a314-b083c06c28c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2"
Nov 03 19:47:09.553098 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.552992    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feaac383-b97d-42ad-b021-2e24d707739b-plugins-config-volume\") pod \"payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l\" (UID: \"feaac383-b97d-42ad-b021-2e24d707739b\") " pod="openshift-ingress/payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l"
Nov 03 19:47:09.553098 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.553032    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-metrics-certs\") pod \"maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb\" (UID: \"b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb"
Nov 03 19:47:09.553098 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.553060    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-maas-api-tls\") pod \"maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb\" (UID: \"b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb"
Nov 03 19:47:09.553338 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.553119    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-openshift-service-ca\") pod \"maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb\" (UID: \"b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb"
Nov 03 19:47:09.553338 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.553266    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d9c6d40-8002-4c1d-a314-b083c06c28c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2\" (UID: \"7d9c6d40-8002-4c1d-a314-b083c06c28c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2"
Nov 03 19:47:09.553700 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.553682    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feaac383-b97d-42ad-b021-2e24d707739b-plugins-config-volume\") pod \"payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l\" (UID: \"feaac383-b97d-42ad-b021-2e24d707739b\") " pod="openshift-ingress/payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l"
Nov 03 19:47:09.553922 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.553906    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d9c6d40-8002-4c1d-a314-b083c06c28c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2\" (UID: \"7d9c6d40-8002-4c1d-a314-b083c06c28c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2"
Nov 03 19:47:09.570265 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.570243    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sppsg\" (UniqueName: \"kubernetes.io/projected/7d9c6d40-8002-4c1d-a314-b083c06c28c1-kube-api-access-sppsg\") pod \"payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2\" (UID: \"7d9c6d40-8002-4c1d-a314-b083c06c28c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2"
Nov 03 19:47:09.570403 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.570388    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wq74\" (UniqueName: \"kubernetes.io/projected/feaac383-b97d-42ad-b021-2e24d707739b-kube-api-access-6wq74\") pod \"payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l\" (UID: \"feaac383-b97d-42ad-b021-2e24d707739b\") " pod="openshift-ingress/payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l"
Nov 03 19:47:09.637415 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.637375    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l"
Nov 03 19:47:09.640392 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.640357    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2"
Nov 03 19:47:09.654683 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.654655    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-metrics-certs\") pod \"maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb\" (UID: \"b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb"
Nov 03 19:47:09.654789 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.654695    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-maas-api-tls\") pod \"maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb\" (UID: \"b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb"
Nov 03 19:47:09.654789 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.654717    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-openshift-service-ca\") pod \"maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb\" (UID: \"b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb"
Nov 03 19:47:09.654897 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.654807    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssmlc\" (UniqueName: \"kubernetes.io/projected/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-kube-api-access-ssmlc\") pod \"maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb\" (UID: \"b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb"
Nov 03 19:47:09.654897 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:47:09.654873    5591 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-serving-cert: secret "maas-api-e2e-ait-c20ed791-serving-cert" not found
Nov 03 19:47:09.655004 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:47:09.654969    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-maas-api-tls podName:b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc nodeName:}" failed. No retries permitted until 2026-11-03 19:47:10.154944268 +0000 UTC m=+578.927237278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-maas-api-tls") pod "maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb" (UID: "b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc") : secret "maas-api-e2e-ait-c20ed791-serving-cert" not found
Nov 03 19:47:09.655553 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.655531    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-openshift-service-ca\") pod \"maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb\" (UID: \"b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb"
Nov 03 19:47:09.657152 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.657134    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-metrics-certs\") pod \"maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb\" (UID: \"b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb"
Nov 03 19:47:09.667727 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.667704    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssmlc\" (UniqueName: \"kubernetes.io/projected/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-kube-api-access-ssmlc\") pod \"maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb\" (UID: \"b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb"
Nov 03 19:47:09.821705 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.821649    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7" event={"ID":"5b020e42-25c7-41b6-93fa-ebc2749579a4","Type":"ContainerStarted","Data":"6fc68052e7c5ecb72b8f5134017646d7d02df0c7c54b4d642cc2509c9f28d6d3"}
Nov 03 19:47:09.821902 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.821715    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7"
Nov 03 19:47:09.821902 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.821739    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7" event={"ID":"5b020e42-25c7-41b6-93fa-ebc2749579a4","Type":"ContainerStarted","Data":"ac7f3046d9512d4b01c1432d57cd0bd8bb85fca54587e5d80deee6f47e87765e"}
Nov 03 19:47:09.823349 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.823317    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d" event={"ID":"41174c3b-6b28-4a8c-82e8-acbf40ba454f","Type":"ContainerStarted","Data":"29f78e2cc006f78a5e9659e7a5ccd435eca4db7e5ce7983f655e13b7419be0d6"}
Nov 03 19:47:09.823471 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.823373    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d" event={"ID":"41174c3b-6b28-4a8c-82e8-acbf40ba454f","Type":"ContainerStarted","Data":"d6c5a8e171a37903112deb01de937685f48e39e6954d298fba329bc88125c8cc"}
Nov 03 19:47:09.823511 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.823493    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:09.823558 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.823528    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d"
Nov 03 19:47:09.824401 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.824382    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:09.853519 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.853464    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7" podStartSLOduration=1.853445389 podStartE2EDuration="1.853445389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:09.852696974 +0000 UTC m=+578.624989993" watchObservedRunningTime="2026-11-03 19:47:09.853445389 +0000 UTC m=+578.625738409"
Nov 03 19:47:09.937620 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.937406    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l"]
Nov 03 19:47:09.937995 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.937939    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d" podStartSLOduration=1.937920397 podStartE2EDuration="1.937920397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:09.933395446 +0000 UTC m=+578.705688466" watchObservedRunningTime="2026-11-03 19:47:09.937920397 +0000 UTC m=+578.710213417"
Nov 03 19:47:09.943237 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:47:09.943191    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeaac383_b97d_42ad_b021_2e24d707739b.slice/crio-63e1675b772cf2a28988ef77aeee766b2a452f8c38c4943afacf9be45962379d WatchSource:0}: Error finding container 63e1675b772cf2a28988ef77aeee766b2a452f8c38c4943afacf9be45962379d: Status 404 returned error can't find the container with id 63e1675b772cf2a28988ef77aeee766b2a452f8c38c4943afacf9be45962379d
Nov 03 19:47:09.958377 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:09.958351    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2"]
Nov 03 19:47:09.960989 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:47:09.960960    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d9c6d40_8002_4c1d_a314_b083c06c28c1.slice/crio-0f85db0edb9c1812c776a4a3faded40065ae5e2708e4bf9de7143a85aa6ace69 WatchSource:0}: Error finding container 0f85db0edb9c1812c776a4a3faded40065ae5e2708e4bf9de7143a85aa6ace69: Status 404 returned error can't find the container with id 0f85db0edb9c1812c776a4a3faded40065ae5e2708e4bf9de7143a85aa6ace69
Nov 03 19:47:10.159928 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:10.159424    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-maas-api-tls\") pod \"maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb\" (UID: \"b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb"
Nov 03 19:47:10.165672 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:10.165646    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-maas-api-tls\") pod \"maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb\" (UID: \"b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb"
Nov 03 19:47:10.391856 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:10.391821    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb"
Nov 03 19:47:10.718053 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:10.717981    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb"]
Nov 03 19:47:10.721359 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:47:10.721332    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ae0fa5_a3fd_445d_b3e8_f1b71d4e3afc.slice/crio-5dc59ec0d0e3f8bf2a61264ac060b0dc636af5f719f246cc38777b24d01e9110 WatchSource:0}: Error finding container 5dc59ec0d0e3f8bf2a61264ac060b0dc636af5f719f246cc38777b24d01e9110: Status 404 returned error can't find the container with id 5dc59ec0d0e3f8bf2a61264ac060b0dc636af5f719f246cc38777b24d01e9110
Nov 03 19:47:10.830135 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:10.830094    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2" event={"ID":"7d9c6d40-8002-4c1d-a314-b083c06c28c1","Type":"ContainerStarted","Data":"316deba7884bdc24dfef20ea63cab2a51034d414bcc68c2c81987bbd36924cf8"}
Nov 03 19:47:10.830344 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:10.830171    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2" event={"ID":"7d9c6d40-8002-4c1d-a314-b083c06c28c1","Type":"ContainerStarted","Data":"0f85db0edb9c1812c776a4a3faded40065ae5e2708e4bf9de7143a85aa6ace69"}
Nov 03 19:47:10.830344 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:10.830320    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2"
Nov 03 19:47:10.831586 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:10.831549    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb" event={"ID":"b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc","Type":"ContainerStarted","Data":"5dc59ec0d0e3f8bf2a61264ac060b0dc636af5f719f246cc38777b24d01e9110"}
Nov 03 19:47:10.833181 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:10.833145    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l" event={"ID":"feaac383-b97d-42ad-b021-2e24d707739b","Type":"ContainerStarted","Data":"1f6e365df77b8abaaaccbade055e616cd3aa6a9821c64533eb409900bb321cff"}
Nov 03 19:47:10.833324 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:10.833227    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l" event={"ID":"feaac383-b97d-42ad-b021-2e24d707739b","Type":"ContainerStarted","Data":"63e1675b772cf2a28988ef77aeee766b2a452f8c38c4943afacf9be45962379d"}
Nov 03 19:47:10.833759 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:10.833738    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l"
Nov 03 19:47:10.849889 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:10.849847    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2" podStartSLOduration=1.849833384 podStartE2EDuration="1.849833384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:10.848272708 +0000 UTC m=+579.620565728" watchObservedRunningTime="2026-11-03 19:47:10.849833384 +0000 UTC m=+579.622126403"
Nov 03 19:47:10.867564 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:10.867517    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l" podStartSLOduration=1.867503949 podStartE2EDuration="1.867503949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:10.866842479 +0000 UTC m=+579.639135495" watchObservedRunningTime="2026-11-03 19:47:10.867503949 +0000 UTC m=+579.639797014"
Nov 03 19:47:11.839679 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:11.839634    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb" event={"ID":"b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc","Type":"ContainerStarted","Data":"de2c2b0ebcfdb902c099ce2f828c450a37e52454bb1a4a9ceddee506113e8dd1"}
Nov 03 19:47:11.840089 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:11.839833    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb"
Nov 03 19:47:11.861306 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:11.861258    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb" podStartSLOduration=2.326012958 podStartE2EDuration="2.861244856s" totalImagesPullingTime="535.231898ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:11.859589878 +0000 UTC m=+580.631882892" watchObservedRunningTime="2026-11-03 19:47:11.861244856 +0000 UTC m=+580.633537873"
Nov 03 19:47:12.149079 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:12.149003    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:47:12.639718 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:12.639686    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh"]
Nov 03 19:47:12.643748 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:12.643724    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh"
Nov 03 19:47:12.651296 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:12.651243    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-9959e5f4-serving-cert\""
Nov 03 19:47:12.655597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:12.655573    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh"]
Nov 03 19:47:12.803185 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:12.803138    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-maas-api-tls\") pod \"maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh\" (UID: \"2240b7e2-d7e7-49e0-b41c-1a9781ce190b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh"
Nov 03 19:47:12.803436 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:12.803262    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh7pm\" (UniqueName: \"kubernetes.io/projected/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-kube-api-access-qh7pm\") pod \"maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh\" (UID: \"2240b7e2-d7e7-49e0-b41c-1a9781ce190b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh"
Nov 03 19:47:12.803436 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:12.803306    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-metrics-certs\") pod \"maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh\" (UID: \"2240b7e2-d7e7-49e0-b41c-1a9781ce190b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh"
Nov 03 19:47:12.803436 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:12.803357    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-openshift-service-ca\") pod \"maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh\" (UID: \"2240b7e2-d7e7-49e0-b41c-1a9781ce190b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh"
Nov 03 19:47:12.905057 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:12.904968    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-maas-api-tls\") pod \"maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh\" (UID: \"2240b7e2-d7e7-49e0-b41c-1a9781ce190b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh"
Nov 03 19:47:12.905057 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:12.905030    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh7pm\" (UniqueName: \"kubernetes.io/projected/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-kube-api-access-qh7pm\") pod \"maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh\" (UID: \"2240b7e2-d7e7-49e0-b41c-1a9781ce190b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh"
Nov 03 19:47:12.905057 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:12.905052    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-metrics-certs\") pod \"maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh\" (UID: \"2240b7e2-d7e7-49e0-b41c-1a9781ce190b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh"
Nov 03 19:47:12.905560 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:12.905079    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-openshift-service-ca\") pod \"maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh\" (UID: \"2240b7e2-d7e7-49e0-b41c-1a9781ce190b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh"
Nov 03 19:47:12.905827 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:12.905806    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-openshift-service-ca\") pod \"maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh\" (UID: \"2240b7e2-d7e7-49e0-b41c-1a9781ce190b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh"
Nov 03 19:47:12.907644 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:12.907624    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-metrics-certs\") pod \"maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh\" (UID: \"2240b7e2-d7e7-49e0-b41c-1a9781ce190b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh"
Nov 03 19:47:12.907699 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:12.907683    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-maas-api-tls\") pod \"maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh\" (UID: \"2240b7e2-d7e7-49e0-b41c-1a9781ce190b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh"
Nov 03 19:47:12.917291 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:12.917264    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh7pm\" (UniqueName: \"kubernetes.io/projected/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-kube-api-access-qh7pm\") pod \"maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh\" (UID: \"2240b7e2-d7e7-49e0-b41c-1a9781ce190b\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh"
Nov 03 19:47:12.962761 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:12.962729    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh"
Nov 03 19:47:13.245384 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:13.245357    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh"]
Nov 03 19:47:13.247928 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:47:13.247901    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2240b7e2_d7e7_49e0_b41c_1a9781ce190b.slice/crio-87e7a943e8784fd4515aa31252dcf62707127f0d4ae23807757f34062eca5ecb WatchSource:0}: Error finding container 87e7a943e8784fd4515aa31252dcf62707127f0d4ae23807757f34062eca5ecb: Status 404 returned error can't find the container with id 87e7a943e8784fd4515aa31252dcf62707127f0d4ae23807757f34062eca5ecb
Nov 03 19:47:13.853000 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:13.852971    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh" event={"ID":"2240b7e2-d7e7-49e0-b41c-1a9781ce190b","Type":"ContainerStarted","Data":"87e7a943e8784fd4515aa31252dcf62707127f0d4ae23807757f34062eca5ecb"}
Nov 03 19:47:14.860749 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:14.860706    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh" event={"ID":"2240b7e2-d7e7-49e0-b41c-1a9781ce190b","Type":"ContainerStarted","Data":"78beb7ed0c9e73e44969ec9f6615de41c0e6e5733e2a3e360bab45e34c4d07a8"}
Nov 03 19:47:14.861178 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:14.860803    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh"
Nov 03 19:47:17.833829 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:17.833758    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh" podStartSLOduration=5.316822549 podStartE2EDuration="5.833740023s" totalImagesPullingTime="516.917474ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:14.893527506 +0000 UTC m=+583.665820519" watchObservedRunningTime="2026-11-03 19:47:17.833740023 +0000 UTC m=+586.606033042"
Nov 03 19:47:17.834548 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:17.834525    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:47:17.852131 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:17.852089    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb"
Nov 03 19:47:20.836298 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:20.836270    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7"
Nov 03 19:47:20.836867 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:20.836523    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d"
Nov 03 19:47:20.872358 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:20.872331    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh"
Nov 03 19:47:21.843004 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:21.842976    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2"
Nov 03 19:47:22.615573 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:22.615541    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:47:22.848372 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:22.848334    5591 patch_prober.go:29] interesting pod/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:47:22.848894 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:22.848410    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f" podUID="c7b7ef8f-17e3-4fe9-a98d-cbd45650f626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:47:22.849518 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:22.849495    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l"
Nov 03 19:47:23.952709 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:23.952622    5591 generic.go:356] "Generic (PLEG): container finished" podID="c7b7ef8f-17e3-4fe9-a98d-cbd45650f626" containerID="07bd397d52b25687427b76006cc0518d1c65812a4f209c3692e3491100e3b05f" exitCode=0
Nov 03 19:47:23.953119 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:23.952704    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f" event={"ID":"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626","Type":"ContainerDied","Data":"07bd397d52b25687427b76006cc0518d1c65812a4f209c3692e3491100e3b05f"}
Nov 03 19:47:23.953158 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:23.953111    5591 scope.go:122] "RemoveContainer" containerID="07bd397d52b25687427b76006cc0518d1c65812a4f209c3692e3491100e3b05f"
Nov 03 19:47:23.954370 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:23.954341    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:47:23.954437 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:23.954411    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:47:23.954473 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:23.954444    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:47:24.960606 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:24.960572    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f" event={"ID":"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626","Type":"ContainerStarted","Data":"9b593ef82c96e11ceb517d99ed48467d3d3b6dd7ffc592b7b292c2bdb518c6ec"}
Nov 03 19:47:25.196236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:25.195560    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:25.196236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:25.195685    5591 patch_prober.go:29] interesting pod/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 19:47:25.196236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:25.195727    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f" podUID="c7b7ef8f-17e3-4fe9-a98d-cbd45650f626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 19:47:26.203064 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:26.202615    5591 patch_prober.go:29] interesting pod/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 19:47:26.203064 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:26.202677    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f" podUID="c7b7ef8f-17e3-4fe9-a98d-cbd45650f626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 19:47:27.195700 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:27.195657    5591 patch_prober.go:29] interesting pod/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 19:47:27.196038 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:27.195995    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f" podUID="c7b7ef8f-17e3-4fe9-a98d-cbd45650f626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 19:47:28.206673 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:28.206634    5591 patch_prober.go:29] interesting pod/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 19:47:28.207323 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:28.207286    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f" podUID="c7b7ef8f-17e3-4fe9-a98d-cbd45650f626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 19:47:28.739596 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:47:28.739552    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:47:29.199148 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:29.198702    5591 patch_prober.go:29] interesting pod/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 19:47:29.199148 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:29.198767    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f" podUID="c7b7ef8f-17e3-4fe9-a98d-cbd45650f626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 19:47:30.014363 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:30.014319    5591 generic.go:356] "Generic (PLEG): container finished" podID="27e0a187-19d5-45cb-bdd6-711f30368cca" containerID="f21e61b950bb814337c21e1a66b772f692528552543377abd1677bf686013934" exitCode=0
Nov 03 19:47:30.014789 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:30.014390    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr" event={"ID":"27e0a187-19d5-45cb-bdd6-711f30368cca","Type":"ContainerDied","Data":"f21e61b950bb814337c21e1a66b772f692528552543377abd1677bf686013934"}
Nov 03 19:47:30.014832 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:30.014805    5591 scope.go:122] "RemoveContainer" containerID="f21e61b950bb814337c21e1a66b772f692528552543377abd1677bf686013934"
Nov 03 19:47:30.016130 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:30.016099    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:47:30.016198 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:30.016174    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:47:30.016258 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:30.016234    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:47:30.297501 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:30.297415    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:30.298568 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:30.298546    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:30.300789 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:30.300761    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:30.317696 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:30.317662    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b"]
Nov 03 19:47:30.321319 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:30.321284    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b"
Nov 03 19:47:30.326784 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:30.326712    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\""
Nov 03 19:47:30.326784 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:30.326755    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:47:30.334992 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:30.334966    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b"]
Nov 03 19:47:30.456299 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:30.456265    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3587641-fcd3-488f-b8a9-8e77dc90f934-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b\" (UID: \"a3587641-fcd3-488f-b8a9-8e77dc90f934\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b"
Nov 03 19:47:30.456471 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:30.456328    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3587641-fcd3-488f-b8a9-8e77dc90f934-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b\" (UID: \"a3587641-fcd3-488f-b8a9-8e77dc90f934\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b"
Nov 03 19:47:30.456471 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:30.456441    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6cd6\" (UniqueName: \"kubernetes.io/projected/a3587641-fcd3-488f-b8a9-8e77dc90f934-kube-api-access-g6cd6\") pod \"maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b\" (UID: \"a3587641-fcd3-488f-b8a9-8e77dc90f934\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b"
Nov 03 19:47:30.557513 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:30.557432    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3587641-fcd3-488f-b8a9-8e77dc90f934-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b\" (UID: \"a3587641-fcd3-488f-b8a9-8e77dc90f934\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b"
Nov 03 19:47:30.557513 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:30.557491    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3587641-fcd3-488f-b8a9-8e77dc90f934-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b\" (UID: \"a3587641-fcd3-488f-b8a9-8e77dc90f934\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b"
Nov 03 19:47:30.557704 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:30.557524    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6cd6\" (UniqueName: \"kubernetes.io/projected/a3587641-fcd3-488f-b8a9-8e77dc90f934-kube-api-access-g6cd6\") pod \"maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b\" (UID: \"a3587641-fcd3-488f-b8a9-8e77dc90f934\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b"
Nov 03 19:47:30.558230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:30.558180    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3587641-fcd3-488f-b8a9-8e77dc90f934-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b\" (UID: \"a3587641-fcd3-488f-b8a9-8e77dc90f934\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b"
Nov 03 19:47:30.560026 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:30.560007    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3587641-fcd3-488f-b8a9-8e77dc90f934-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b\" (UID: \"a3587641-fcd3-488f-b8a9-8e77dc90f934\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b"
Nov 03 19:47:30.569864 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:30.569839    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6cd6\" (UniqueName: \"kubernetes.io/projected/a3587641-fcd3-488f-b8a9-8e77dc90f934-kube-api-access-g6cd6\") pod \"maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b\" (UID: \"a3587641-fcd3-488f-b8a9-8e77dc90f934\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b"
Nov 03 19:47:30.642399 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:30.642364    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b"
Nov 03 19:47:30.941244 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:30.941183    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b"]
Nov 03 19:47:30.943742 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:47:30.943695    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3587641_fcd3_488f_b8a9_8e77dc90f934.slice/crio-4de08841f583cc1d829275b6bba1dea363b51408132bf0c9c25e5b16b7e7c854 WatchSource:0}: Error finding container 4de08841f583cc1d829275b6bba1dea363b51408132bf0c9c25e5b16b7e7c854: Status 404 returned error can't find the container with id 4de08841f583cc1d829275b6bba1dea363b51408132bf0c9c25e5b16b7e7c854
Nov 03 19:47:31.023086 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:31.023046    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr" event={"ID":"27e0a187-19d5-45cb-bdd6-711f30368cca","Type":"ContainerStarted","Data":"0564a244161a135c39f6148d98cb1c83d1fe40765424aa0286014d3b86be34b2"}
Nov 03 19:47:31.024525 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:31.024497    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b" event={"ID":"a3587641-fcd3-488f-b8a9-8e77dc90f934","Type":"ContainerStarted","Data":"4de08841f583cc1d829275b6bba1dea363b51408132bf0c9c25e5b16b7e7c854"}
Nov 03 19:47:31.446636 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:31.446611    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 19:47:31.448000 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:31.447978    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 19:47:31.931965 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:31.931929    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:31.932457 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:31.932435    5591 patch_prober.go:29] interesting pod/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 19:47:31.932515 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:31.932486    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr" podUID="27e0a187-19d5-45cb-bdd6-711f30368cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 19:47:32.031621 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:32.031585    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b" event={"ID":"a3587641-fcd3-488f-b8a9-8e77dc90f934","Type":"ContainerStarted","Data":"2e2ff41d29f6d5f6ee79a17699b33af1b65c6b7ac909888eb0b18e1844f0c23a"}
Nov 03 19:47:32.052552 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:32.052498    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b" podStartSLOduration=2.052483931 podStartE2EDuration="2.052483931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:32.049916814 +0000 UTC m=+600.822209829" watchObservedRunningTime="2026-11-03 19:47:32.052483931 +0000 UTC m=+600.824776958"
Nov 03 19:47:32.932586 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:32.932543    5591 patch_prober.go:29] interesting pod/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 19:47:32.932774 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:32.932598    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr" podUID="27e0a187-19d5-45cb-bdd6-711f30368cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 19:47:33.039375 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:33.039333    5591 generic.go:356] "Generic (PLEG): container finished" podID="a3587641-fcd3-488f-b8a9-8e77dc90f934" containerID="2e2ff41d29f6d5f6ee79a17699b33af1b65c6b7ac909888eb0b18e1844f0c23a" exitCode=0
Nov 03 19:47:33.039846 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:33.039476    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b" event={"ID":"a3587641-fcd3-488f-b8a9-8e77dc90f934","Type":"ContainerDied","Data":"2e2ff41d29f6d5f6ee79a17699b33af1b65c6b7ac909888eb0b18e1844f0c23a"}
Nov 03 19:47:33.932007 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:33.931972    5591 patch_prober.go:29] interesting pod/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 19:47:33.932232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:33.932027    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr" podUID="27e0a187-19d5-45cb-bdd6-711f30368cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 19:47:34.253064 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:34.253039    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b"
Nov 03 19:47:34.428193 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:34.428159    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3587641-fcd3-488f-b8a9-8e77dc90f934-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3587641-fcd3-488f-b8a9-8e77dc90f934-maas-api-openshift-ca\") pod \"a3587641-fcd3-488f-b8a9-8e77dc90f934\" (UID: \"a3587641-fcd3-488f-b8a9-8e77dc90f934\") "
Nov 03 19:47:34.428390 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:34.428246    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3587641-fcd3-488f-b8a9-8e77dc90f934-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3587641-fcd3-488f-b8a9-8e77dc90f934-maas-api-xks-ca\") pod \"a3587641-fcd3-488f-b8a9-8e77dc90f934\" (UID: \"a3587641-fcd3-488f-b8a9-8e77dc90f934\") "
Nov 03 19:47:34.428390 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:34.428320    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3587641-fcd3-488f-b8a9-8e77dc90f934-kube-api-access-g6cd6\" (UniqueName: \"kubernetes.io/projected/a3587641-fcd3-488f-b8a9-8e77dc90f934-kube-api-access-g6cd6\") pod \"a3587641-fcd3-488f-b8a9-8e77dc90f934\" (UID: \"a3587641-fcd3-488f-b8a9-8e77dc90f934\") "
Nov 03 19:47:34.429104 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:34.429083    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3587641-fcd3-488f-b8a9-8e77dc90f934-maas-api-openshift-ca" pod "a3587641-fcd3-488f-b8a9-8e77dc90f934" (UID: "a3587641-fcd3-488f-b8a9-8e77dc90f934"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:34.433545 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:34.433513    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3587641-fcd3-488f-b8a9-8e77dc90f934-kube-api-access-g6cd6" pod "a3587641-fcd3-488f-b8a9-8e77dc90f934" (UID: "a3587641-fcd3-488f-b8a9-8e77dc90f934"). InnerVolumeSpecName "kube-api-access-g6cd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:34.433545 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:34.433523    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3587641-fcd3-488f-b8a9-8e77dc90f934-maas-api-xks-ca" pod "a3587641-fcd3-488f-b8a9-8e77dc90f934" (UID: "a3587641-fcd3-488f-b8a9-8e77dc90f934"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:34.529182 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:34.529148    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3587641-fcd3-488f-b8a9-8e77dc90f934-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:34.529182 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:34.529177    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6cd6\" (UniqueName: \"kubernetes.io/projected/a3587641-fcd3-488f-b8a9-8e77dc90f934-kube-api-access-g6cd6\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:34.529182 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:34.529188    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3587641-fcd3-488f-b8a9-8e77dc90f934-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:34.933043 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:34.932950    5591 patch_prober.go:29] interesting pod/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 19:47:34.933043 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:34.933021    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr" podUID="27e0a187-19d5-45cb-bdd6-711f30368cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 19:47:35.051568 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:35.051532    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b" event={"ID":"a3587641-fcd3-488f-b8a9-8e77dc90f934","Type":"ContainerDied","Data":"4de08841f583cc1d829275b6bba1dea363b51408132bf0c9c25e5b16b7e7c854"}
Nov 03 19:47:35.051741 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:35.051590    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4de08841f583cc1d829275b6bba1dea363b51408132bf0c9c25e5b16b7e7c854"
Nov 03 19:47:35.051741 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:35.051644    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b"
Nov 03 19:47:35.932260 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:35.932226    5591 patch_prober.go:29] interesting pod/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 19:47:35.932638 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:35.932281    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr" podUID="27e0a187-19d5-45cb-bdd6-711f30368cca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 19:47:36.402246 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:36.402186    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9"]
Nov 03 19:47:36.402990 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:36.402965    5591 cpu_manager.go:414] "containerMap: removing container" podUID="a3587641-fcd3-488f-b8a9-8e77dc90f934" containerName="revoke-keys"
Nov 03 19:47:36.403039 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:36.402994    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3587641-fcd3-488f-b8a9-8e77dc90f934" containerName="revoke-keys"
Nov 03 19:47:36.403133 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:36.403119    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3587641-fcd3-488f-b8a9-8e77dc90f934" containerName="revoke-keys"
Nov 03 19:47:36.405900 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:36.405879    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9"
Nov 03 19:47:36.412503 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:36.412480    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:47:36.412632 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:36.412544    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\""
Nov 03 19:47:36.415279 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:36.415250    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9"]
Nov 03 19:47:36.549504 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:36.549468    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab1a8953-14f8-4983-a423-fa416f9ec7ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9\" (UID: \"ab1a8953-14f8-4983-a423-fa416f9ec7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9"
Nov 03 19:47:36.549704 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:36.549536    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab1a8953-14f8-4983-a423-fa416f9ec7ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9\" (UID: \"ab1a8953-14f8-4983-a423-fa416f9ec7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9"
Nov 03 19:47:36.549704 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:36.549650    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5btnl\" (UniqueName: \"kubernetes.io/projected/ab1a8953-14f8-4983-a423-fa416f9ec7ae-kube-api-access-5btnl\") pod \"maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9\" (UID: \"ab1a8953-14f8-4983-a423-fa416f9ec7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9"
Nov 03 19:47:36.650429 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:36.650387    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab1a8953-14f8-4983-a423-fa416f9ec7ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9\" (UID: \"ab1a8953-14f8-4983-a423-fa416f9ec7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9"
Nov 03 19:47:36.650625 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:36.650483    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5btnl\" (UniqueName: \"kubernetes.io/projected/ab1a8953-14f8-4983-a423-fa416f9ec7ae-kube-api-access-5btnl\") pod \"maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9\" (UID: \"ab1a8953-14f8-4983-a423-fa416f9ec7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9"
Nov 03 19:47:36.650698 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:36.650623    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab1a8953-14f8-4983-a423-fa416f9ec7ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9\" (UID: \"ab1a8953-14f8-4983-a423-fa416f9ec7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9"
Nov 03 19:47:36.651103 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:36.651083    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab1a8953-14f8-4983-a423-fa416f9ec7ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9\" (UID: \"ab1a8953-14f8-4983-a423-fa416f9ec7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9"
Nov 03 19:47:36.653125 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:36.653068    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab1a8953-14f8-4983-a423-fa416f9ec7ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9\" (UID: \"ab1a8953-14f8-4983-a423-fa416f9ec7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9"
Nov 03 19:47:36.663258 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:36.663238    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5btnl\" (UniqueName: \"kubernetes.io/projected/ab1a8953-14f8-4983-a423-fa416f9ec7ae-kube-api-access-5btnl\") pod \"maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9\" (UID: \"ab1a8953-14f8-4983-a423-fa416f9ec7ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9"
Nov 03 19:47:36.719649 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:36.719618    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9"
Nov 03 19:47:36.936573 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:36.936493    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:36.936927 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:36.936805    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:36.937575 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:36.937562    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:37.010142 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:37.010110    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9"]
Nov 03 19:47:37.063002 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:37.062962    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9" event={"ID":"ab1a8953-14f8-4983-a423-fa416f9ec7ae","Type":"ContainerStarted","Data":"0961f09fc303bba40f102ceb78e4fd396cd9fc19f298dd63d64904e2c2d2b4ac"}
Nov 03 19:47:38.071688 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:38.071609    5591 generic.go:356] "Generic (PLEG): container finished" podID="ab1a8953-14f8-4983-a423-fa416f9ec7ae" containerID="fcaa32b0cc3823c4464c0d4c473e0c57742a070f379daf7daf8630a99cebcf3f" exitCode=0
Nov 03 19:47:38.072046 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:38.071687    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9" event={"ID":"ab1a8953-14f8-4983-a423-fa416f9ec7ae","Type":"ContainerDied","Data":"fcaa32b0cc3823c4464c0d4c473e0c57742a070f379daf7daf8630a99cebcf3f"}
Nov 03 19:47:38.262482 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:38.262451    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l"]
Nov 03 19:47:38.280819 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:38.280781    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2"]
Nov 03 19:47:39.243024 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:39.242995    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9"
Nov 03 19:47:39.384039 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:39.383964    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab1a8953-14f8-4983-a423-fa416f9ec7ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab1a8953-14f8-4983-a423-fa416f9ec7ae-maas-api-xks-ca\") pod \"ab1a8953-14f8-4983-a423-fa416f9ec7ae\" (UID: \"ab1a8953-14f8-4983-a423-fa416f9ec7ae\") "
Nov 03 19:47:39.384039 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:39.384014    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab1a8953-14f8-4983-a423-fa416f9ec7ae-kube-api-access-5btnl\" (UniqueName: \"kubernetes.io/projected/ab1a8953-14f8-4983-a423-fa416f9ec7ae-kube-api-access-5btnl\") pod \"ab1a8953-14f8-4983-a423-fa416f9ec7ae\" (UID: \"ab1a8953-14f8-4983-a423-fa416f9ec7ae\") "
Nov 03 19:47:39.384262 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:39.384141    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab1a8953-14f8-4983-a423-fa416f9ec7ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab1a8953-14f8-4983-a423-fa416f9ec7ae-maas-api-openshift-ca\") pod \"ab1a8953-14f8-4983-a423-fa416f9ec7ae\" (UID: \"ab1a8953-14f8-4983-a423-fa416f9ec7ae\") "
Nov 03 19:47:39.384504 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:39.384478    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab1a8953-14f8-4983-a423-fa416f9ec7ae-maas-api-openshift-ca" pod "ab1a8953-14f8-4983-a423-fa416f9ec7ae" (UID: "ab1a8953-14f8-4983-a423-fa416f9ec7ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:39.384636 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:39.384619    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab1a8953-14f8-4983-a423-fa416f9ec7ae-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:39.386357 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:39.386329    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab1a8953-14f8-4983-a423-fa416f9ec7ae-maas-api-xks-ca" pod "ab1a8953-14f8-4983-a423-fa416f9ec7ae" (UID: "ab1a8953-14f8-4983-a423-fa416f9ec7ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:39.386451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:39.386390    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1a8953-14f8-4983-a423-fa416f9ec7ae-kube-api-access-5btnl" pod "ab1a8953-14f8-4983-a423-fa416f9ec7ae" (UID: "ab1a8953-14f8-4983-a423-fa416f9ec7ae"). InnerVolumeSpecName "kube-api-access-5btnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:39.485966 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:39.485931    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab1a8953-14f8-4983-a423-fa416f9ec7ae-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:39.485966 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:39.485968    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5btnl\" (UniqueName: \"kubernetes.io/projected/ab1a8953-14f8-4983-a423-fa416f9ec7ae-kube-api-access-5btnl\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:40.098091 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:40.098050    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9" event={"ID":"ab1a8953-14f8-4983-a423-fa416f9ec7ae","Type":"ContainerDied","Data":"0961f09fc303bba40f102ceb78e4fd396cd9fc19f298dd63d64904e2c2d2b4ac"}
Nov 03 19:47:40.098091 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:40.098080    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9"
Nov 03 19:47:40.098365 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:40.098111    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0961f09fc303bba40f102ceb78e4fd396cd9fc19f298dd63d64904e2c2d2b4ac"
Nov 03 19:47:42.422029 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:42.421995    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh"]
Nov 03 19:47:42.422557 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:42.422239    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh" podUID="2240b7e2-d7e7-49e0-b41c-1a9781ce190b" containerName="maas-api" containerID="cri-o://78beb7ed0c9e73e44969ec9f6615de41c0e6e5733e2a3e360bab45e34c4d07a8" gracePeriod=30
Nov 03 19:47:42.742469 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:42.742438    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh"
Nov 03 19:47:42.843555 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:42.843518    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-kube-api-access-qh7pm\" (UniqueName: \"kubernetes.io/projected/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-kube-api-access-qh7pm\") pod \"2240b7e2-d7e7-49e0-b41c-1a9781ce190b\" (UID: \"2240b7e2-d7e7-49e0-b41c-1a9781ce190b\") "
Nov 03 19:47:42.843736 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:42.843592    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-maas-api-tls\") pod \"2240b7e2-d7e7-49e0-b41c-1a9781ce190b\" (UID: \"2240b7e2-d7e7-49e0-b41c-1a9781ce190b\") "
Nov 03 19:47:42.843736 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:42.843627    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-metrics-certs\") pod \"2240b7e2-d7e7-49e0-b41c-1a9781ce190b\" (UID: \"2240b7e2-d7e7-49e0-b41c-1a9781ce190b\") "
Nov 03 19:47:42.843736 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:42.843651    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-openshift-service-ca\") pod \"2240b7e2-d7e7-49e0-b41c-1a9781ce190b\" (UID: \"2240b7e2-d7e7-49e0-b41c-1a9781ce190b\") "
Nov 03 19:47:42.844112 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:42.844080    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-openshift-service-ca" pod "2240b7e2-d7e7-49e0-b41c-1a9781ce190b" (UID: "2240b7e2-d7e7-49e0-b41c-1a9781ce190b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:42.845750 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:42.845717    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-metrics-certs" pod "2240b7e2-d7e7-49e0-b41c-1a9781ce190b" (UID: "2240b7e2-d7e7-49e0-b41c-1a9781ce190b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:42.845855 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:42.845784    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-maas-api-tls" pod "2240b7e2-d7e7-49e0-b41c-1a9781ce190b" (UID: "2240b7e2-d7e7-49e0-b41c-1a9781ce190b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:42.845855 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:42.845799    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-kube-api-access-qh7pm" pod "2240b7e2-d7e7-49e0-b41c-1a9781ce190b" (UID: "2240b7e2-d7e7-49e0-b41c-1a9781ce190b"). InnerVolumeSpecName "kube-api-access-qh7pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:42.944913 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:42.944878    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh7pm\" (UniqueName: \"kubernetes.io/projected/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-kube-api-access-qh7pm\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:42.944913 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:42.944908    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:42.945120 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:42.944923    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:42.945120 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:42.944938    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2240b7e2-d7e7-49e0-b41c-1a9781ce190b-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:43.114055 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.114015    5591 generic.go:356] "Generic (PLEG): container finished" podID="2240b7e2-d7e7-49e0-b41c-1a9781ce190b" containerID="78beb7ed0c9e73e44969ec9f6615de41c0e6e5733e2a3e360bab45e34c4d07a8" exitCode=0
Nov 03 19:47:43.114240 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.114064    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh"
Nov 03 19:47:43.114240 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.114079    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh" event={"ID":"2240b7e2-d7e7-49e0-b41c-1a9781ce190b","Type":"ContainerDied","Data":"78beb7ed0c9e73e44969ec9f6615de41c0e6e5733e2a3e360bab45e34c4d07a8"}
Nov 03 19:47:43.114240 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.114137    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh" event={"ID":"2240b7e2-d7e7-49e0-b41c-1a9781ce190b","Type":"ContainerDied","Data":"87e7a943e8784fd4515aa31252dcf62707127f0d4ae23807757f34062eca5ecb"}
Nov 03 19:47:43.114240 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.114168    5591 scope.go:122] "RemoveContainer" containerID="78beb7ed0c9e73e44969ec9f6615de41c0e6e5733e2a3e360bab45e34c4d07a8"
Nov 03 19:47:43.126364 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.126342    5591 scope.go:122] "RemoveContainer" containerID="78beb7ed0c9e73e44969ec9f6615de41c0e6e5733e2a3e360bab45e34c4d07a8"
Nov 03 19:47:43.126776 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:47:43.126753    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78beb7ed0c9e73e44969ec9f6615de41c0e6e5733e2a3e360bab45e34c4d07a8\": container with ID starting with 78beb7ed0c9e73e44969ec9f6615de41c0e6e5733e2a3e360bab45e34c4d07a8 not found: ID does not exist" containerID="78beb7ed0c9e73e44969ec9f6615de41c0e6e5733e2a3e360bab45e34c4d07a8"
Nov 03 19:47:43.126929 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.126784    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78beb7ed0c9e73e44969ec9f6615de41c0e6e5733e2a3e360bab45e34c4d07a8"} err="failed to get container status \"78beb7ed0c9e73e44969ec9f6615de41c0e6e5733e2a3e360bab45e34c4d07a8\": rpc error: code = NotFound desc = could not find container \"78beb7ed0c9e73e44969ec9f6615de41c0e6e5733e2a3e360bab45e34c4d07a8\": container with ID starting with 78beb7ed0c9e73e44969ec9f6615de41c0e6e5733e2a3e360bab45e34c4d07a8 not found: ID does not exist"
Nov 03 19:47:43.147932 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.147898    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh"]
Nov 03 19:47:43.149660 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.149637    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9959e5f4-84678946dc-wcxbh"]
Nov 03 19:47:43.158253 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:47:43.158189    5591 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2240b7e2_d7e7_49e0_b41c_1a9781ce190b.slice/crio-87e7a943e8784fd4515aa31252dcf62707127f0d4ae23807757f34062eca5ecb\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2240b7e2_d7e7_49e0_b41c_1a9781ce190b.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 19:47:43.263687 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.263614    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l" podUID="feaac383-b97d-42ad-b021-2e24d707739b" containerName="payload-processing" containerID="cri-o://1f6e365df77b8abaaaccbade055e616cd3aa6a9821c64533eb409900bb321cff" gracePeriod=25
Nov 03 19:47:43.281338 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.281307    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2" podUID="7d9c6d40-8002-4c1d-a314-b083c06c28c1" containerName="payload-pre-processing" containerID="cri-o://316deba7884bdc24dfef20ea63cab2a51034d414bcc68c2c81987bbd36924cf8" gracePeriod=25
Nov 03 19:47:43.347777 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.347749    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2240b7e2-d7e7-49e0-b41c-1a9781ce190b" path="/var/lib/kubelet/pods/2240b7e2-d7e7-49e0-b41c-1a9781ce190b/volumes"
Nov 03 19:47:43.536194 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.536170    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l"
Nov 03 19:47:43.590971 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.590946    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2"
Nov 03 19:47:43.652554 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.652477    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feaac383-b97d-42ad-b021-2e24d707739b-kube-api-access-6wq74\" (UniqueName: \"kubernetes.io/projected/feaac383-b97d-42ad-b021-2e24d707739b-kube-api-access-6wq74\") pod \"feaac383-b97d-42ad-b021-2e24d707739b\" (UID: \"feaac383-b97d-42ad-b021-2e24d707739b\") "
Nov 03 19:47:43.652715 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.652584    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d9c6d40-8002-4c1d-a314-b083c06c28c1-kube-api-access-sppsg\" (UniqueName: \"kubernetes.io/projected/7d9c6d40-8002-4c1d-a314-b083c06c28c1-kube-api-access-sppsg\") pod \"7d9c6d40-8002-4c1d-a314-b083c06c28c1\" (UID: \"7d9c6d40-8002-4c1d-a314-b083c06c28c1\") "
Nov 03 19:47:43.652715 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.652642    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d9c6d40-8002-4c1d-a314-b083c06c28c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d9c6d40-8002-4c1d-a314-b083c06c28c1-plugins-config-volume\") pod \"7d9c6d40-8002-4c1d-a314-b083c06c28c1\" (UID: \"7d9c6d40-8002-4c1d-a314-b083c06c28c1\") "
Nov 03 19:47:43.652837 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.652728    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feaac383-b97d-42ad-b021-2e24d707739b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feaac383-b97d-42ad-b021-2e24d707739b-plugins-config-volume\") pod \"feaac383-b97d-42ad-b021-2e24d707739b\" (UID: \"feaac383-b97d-42ad-b021-2e24d707739b\") "
Nov 03 19:47:43.653163 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.653077    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d9c6d40-8002-4c1d-a314-b083c06c28c1-plugins-config-volume" pod "7d9c6d40-8002-4c1d-a314-b083c06c28c1" (UID: "7d9c6d40-8002-4c1d-a314-b083c06c28c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:43.653469 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.653283    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feaac383-b97d-42ad-b021-2e24d707739b-plugins-config-volume" pod "feaac383-b97d-42ad-b021-2e24d707739b" (UID: "feaac383-b97d-42ad-b021-2e24d707739b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:43.655411 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.655389    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9c6d40-8002-4c1d-a314-b083c06c28c1-kube-api-access-sppsg" pod "7d9c6d40-8002-4c1d-a314-b083c06c28c1" (UID: "7d9c6d40-8002-4c1d-a314-b083c06c28c1"). InnerVolumeSpecName "kube-api-access-sppsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:43.655514 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.655445    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feaac383-b97d-42ad-b021-2e24d707739b-kube-api-access-6wq74" pod "feaac383-b97d-42ad-b021-2e24d707739b" (UID: "feaac383-b97d-42ad-b021-2e24d707739b"). InnerVolumeSpecName "kube-api-access-6wq74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:43.754301 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.754259    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feaac383-b97d-42ad-b021-2e24d707739b-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:43.754301 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.754295    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wq74\" (UniqueName: \"kubernetes.io/projected/feaac383-b97d-42ad-b021-2e24d707739b-kube-api-access-6wq74\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:43.754301 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.754309    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sppsg\" (UniqueName: \"kubernetes.io/projected/7d9c6d40-8002-4c1d-a314-b083c06c28c1-kube-api-access-sppsg\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:43.754519 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.754321    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d9c6d40-8002-4c1d-a314-b083c06c28c1-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:43.807978 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.807947    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7"]
Nov 03 19:47:43.831146 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.831107    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d"]
Nov 03 19:47:43.963106 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.963023    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"]
Nov 03 19:47:43.963799 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.963777    5591 cpu_manager.go:414] "containerMap: removing container" podUID="feaac383-b97d-42ad-b021-2e24d707739b" containerName="payload-processing"
Nov 03 19:47:43.963799 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.963799    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feaac383-b97d-42ad-b021-2e24d707739b" containerName="payload-processing"
Nov 03 19:47:43.963958 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.963814    5591 cpu_manager.go:414] "containerMap: removing container" podUID="ab1a8953-14f8-4983-a423-fa416f9ec7ae" containerName="revoke-keys"
Nov 03 19:47:43.963958 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.963823    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab1a8953-14f8-4983-a423-fa416f9ec7ae" containerName="revoke-keys"
Nov 03 19:47:43.963958 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.963865    5591 cpu_manager.go:414] "containerMap: removing container" podUID="7d9c6d40-8002-4c1d-a314-b083c06c28c1" containerName="payload-pre-processing"
Nov 03 19:47:43.963958 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.963874    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d9c6d40-8002-4c1d-a314-b083c06c28c1" containerName="payload-pre-processing"
Nov 03 19:47:43.963958 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.963887    5591 cpu_manager.go:414] "containerMap: removing container" podUID="2240b7e2-d7e7-49e0-b41c-1a9781ce190b" containerName="maas-api"
Nov 03 19:47:43.963958 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.963895    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2240b7e2-d7e7-49e0-b41c-1a9781ce190b" containerName="maas-api"
Nov 03 19:47:43.964165 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.963985    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d9c6d40-8002-4c1d-a314-b083c06c28c1" containerName="payload-pre-processing"
Nov 03 19:47:43.964165 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.963996    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="2240b7e2-d7e7-49e0-b41c-1a9781ce190b" containerName="maas-api"
Nov 03 19:47:43.964165 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.964010    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="feaac383-b97d-42ad-b021-2e24d707739b" containerName="payload-processing"
Nov 03 19:47:43.964165 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.964019    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab1a8953-14f8-4983-a423-fa416f9ec7ae" containerName="revoke-keys"
Nov 03 19:47:43.967611 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.967589    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:43.971958 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.971935    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-2b5703-openshift-default-dockercfg-jmjbm\""
Nov 03 19:47:43.982350 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:43.982322    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"]
Nov 03 19:47:44.056989 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.056960    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.057145 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.057004    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.057145 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.057077    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.057145 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.057123    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.057268 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.057147    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.057268 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.057185    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.057355 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.057337    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tz2f\" (UniqueName: \"kubernetes.io/projected/1faa21e4-5bf7-418d-adc8-3fc5bf955010-kube-api-access-6tz2f\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.057390 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.057369    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.057423 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.057395    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.121551 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.121516    5591 generic.go:356] "Generic (PLEG): container finished" podID="feaac383-b97d-42ad-b021-2e24d707739b" containerID="1f6e365df77b8abaaaccbade055e616cd3aa6a9821c64533eb409900bb321cff" exitCode=0
Nov 03 19:47:44.121753 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.121595    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l"
Nov 03 19:47:44.121753 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.121603    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l" event={"ID":"feaac383-b97d-42ad-b021-2e24d707739b","Type":"ContainerDied","Data":"1f6e365df77b8abaaaccbade055e616cd3aa6a9821c64533eb409900bb321cff"}
Nov 03 19:47:44.121753 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.121673    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l" event={"ID":"feaac383-b97d-42ad-b021-2e24d707739b","Type":"ContainerDied","Data":"63e1675b772cf2a28988ef77aeee766b2a452f8c38c4943afacf9be45962379d"}
Nov 03 19:47:44.121753 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.121703    5591 scope.go:122] "RemoveContainer" containerID="1f6e365df77b8abaaaccbade055e616cd3aa6a9821c64533eb409900bb321cff"
Nov 03 19:47:44.124314 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.124292    5591 generic.go:356] "Generic (PLEG): container finished" podID="7d9c6d40-8002-4c1d-a314-b083c06c28c1" containerID="316deba7884bdc24dfef20ea63cab2a51034d414bcc68c2c81987bbd36924cf8" exitCode=0
Nov 03 19:47:44.124420 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.124340    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2"
Nov 03 19:47:44.124420 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.124383    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2" event={"ID":"7d9c6d40-8002-4c1d-a314-b083c06c28c1","Type":"ContainerDied","Data":"316deba7884bdc24dfef20ea63cab2a51034d414bcc68c2c81987bbd36924cf8"}
Nov 03 19:47:44.124504 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.124420    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2" event={"ID":"7d9c6d40-8002-4c1d-a314-b083c06c28c1","Type":"ContainerDied","Data":"0f85db0edb9c1812c776a4a3faded40065ae5e2708e4bf9de7143a85aa6ace69"}
Nov 03 19:47:44.132279 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.132188    5591 scope.go:122] "RemoveContainer" containerID="1f6e365df77b8abaaaccbade055e616cd3aa6a9821c64533eb409900bb321cff"
Nov 03 19:47:44.132575 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:47:44.132537    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f6e365df77b8abaaaccbade055e616cd3aa6a9821c64533eb409900bb321cff\": container with ID starting with 1f6e365df77b8abaaaccbade055e616cd3aa6a9821c64533eb409900bb321cff not found: ID does not exist" containerID="1f6e365df77b8abaaaccbade055e616cd3aa6a9821c64533eb409900bb321cff"
Nov 03 19:47:44.132682 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.132576    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f6e365df77b8abaaaccbade055e616cd3aa6a9821c64533eb409900bb321cff"} err="failed to get container status \"1f6e365df77b8abaaaccbade055e616cd3aa6a9821c64533eb409900bb321cff\": rpc error: code = NotFound desc = could not find container \"1f6e365df77b8abaaaccbade055e616cd3aa6a9821c64533eb409900bb321cff\": container with ID starting with 1f6e365df77b8abaaaccbade055e616cd3aa6a9821c64533eb409900bb321cff not found: ID does not exist"
Nov 03 19:47:44.132682 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.132598    5591 scope.go:122] "RemoveContainer" containerID="316deba7884bdc24dfef20ea63cab2a51034d414bcc68c2c81987bbd36924cf8"
Nov 03 19:47:44.141577 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.141563    5591 scope.go:122] "RemoveContainer" containerID="316deba7884bdc24dfef20ea63cab2a51034d414bcc68c2c81987bbd36924cf8"
Nov 03 19:47:44.141849 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:47:44.141825    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"316deba7884bdc24dfef20ea63cab2a51034d414bcc68c2c81987bbd36924cf8\": container with ID starting with 316deba7884bdc24dfef20ea63cab2a51034d414bcc68c2c81987bbd36924cf8 not found: ID does not exist" containerID="316deba7884bdc24dfef20ea63cab2a51034d414bcc68c2c81987bbd36924cf8"
Nov 03 19:47:44.141906 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.141859    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"316deba7884bdc24dfef20ea63cab2a51034d414bcc68c2c81987bbd36924cf8"} err="failed to get container status \"316deba7884bdc24dfef20ea63cab2a51034d414bcc68c2c81987bbd36924cf8\": rpc error: code = NotFound desc = could not find container \"316deba7884bdc24dfef20ea63cab2a51034d414bcc68c2c81987bbd36924cf8\": container with ID starting with 316deba7884bdc24dfef20ea63cab2a51034d414bcc68c2c81987bbd36924cf8 not found: ID does not exist"
Nov 03 19:47:44.151748 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.151676    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l"]
Nov 03 19:47:44.153264 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.153247    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b"]
Nov 03 19:47:44.155940 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.155921    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-745a88f9-59fd8c46cd-k9x2l"]
Nov 03 19:47:44.157965 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.157941    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.158070 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.158001    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tz2f\" (UniqueName: \"kubernetes.io/projected/1faa21e4-5bf7-418d-adc8-3fc5bf955010-kube-api-access-6tz2f\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.158070 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.158021    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.158070 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.158045    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.158258 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.158099    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.158258 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.158146    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.158258 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.158186    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.158258 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.158243    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.158470 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.158275    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.158470 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.158443    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.158656 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.158620    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.158902 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.158883    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.159104 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.159084    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.159771 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.159730    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.159771 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.159742    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3f7e10b8-bb95b"]
Nov 03 19:47:44.161479 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.161453    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.161835 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.161814    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.167633 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.167612    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2"]
Nov 03 19:47:44.169554 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.169536    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-745a88f9-55f698d8f9-2qzd2"]
Nov 03 19:47:44.172636 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.172619    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.172916 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.172900    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tz2f\" (UniqueName: \"kubernetes.io/projected/1faa21e4-5bf7-418d-adc8-3fc5bf955010-kube-api-access-6tz2f\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.287188 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.287153    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:44.301941 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.301864    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"]
Nov 03 19:47:44.306272 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.306245    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.314436 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.314412    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-9f46e5-openshift-default-dockercfg-4djl8\""
Nov 03 19:47:44.319689 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.319666    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"]
Nov 03 19:47:44.362251 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.361085    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.362251 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.361131    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.362251 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.361191    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.362251 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.361255    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.362251 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.361288    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.362251 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.361328    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.362251 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.361396    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.362251 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.361499    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.362251 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.361546    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znbm4\" (UniqueName: \"kubernetes.io/projected/8e00f1a1-8156-4fa7-8308-b48127f82aa6-kube-api-access-znbm4\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.463256 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.462845    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.463256 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.462897    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znbm4\" (UniqueName: \"kubernetes.io/projected/8e00f1a1-8156-4fa7-8308-b48127f82aa6-kube-api-access-znbm4\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.463256 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.462944    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.463256 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.462972    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.463256 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.463027    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.463256 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.463062    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.463256 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.463094    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.463256 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.463130    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.463894 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.463760    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.464025 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.464003    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.464077 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.464018    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.464077 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.464028    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.464410 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.464360    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.464533 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.464464    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.466719 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.466658    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.467636 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.467609    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.477914 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.477873    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.483434 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.483384    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znbm4\" (UniqueName: \"kubernetes.io/projected/8e00f1a1-8156-4fa7-8308-b48127f82aa6-kube-api-access-znbm4\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.591992 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.591906    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"]
Nov 03 19:47:44.604720 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.604679    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:47:44.604917 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.604898    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:47:44.605048 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.605031    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:47:44.628110 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.628087    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:44.803473 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.803389    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"]
Nov 03 19:47:44.804013 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.803982    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr" podUID="27e0a187-19d5-45cb-bdd6-711f30368cca" containerName="istio-proxy" containerID="cri-o://0564a244161a135c39f6148d98cb1c83d1fe40765424aa0286014d3b86be34b2" gracePeriod=30
Nov 03 19:47:44.930770 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.929314    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"]
Nov 03 19:47:44.936861 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:47:44.936817    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e00f1a1_8156_4fa7_8308_b48127f82aa6.slice/crio-c80c5955640014d9c6083295d31e98784534ca733a4f90f8331c64acf63db028 WatchSource:0}: Error finding container c80c5955640014d9c6083295d31e98784534ca733a4f90f8331c64acf63db028: Status 404 returned error can't find the container with id c80c5955640014d9c6083295d31e98784534ca733a4f90f8331c64acf63db028
Nov 03 19:47:44.939972 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.939930    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:47:44.940063 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.940026    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:47:44.940109 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:44.940073    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:47:45.133801 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:45.133762    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r" event={"ID":"8e00f1a1-8156-4fa7-8308-b48127f82aa6","Type":"ContainerStarted","Data":"da60d0a13ec638f966a0e24803738a66382b122c2968c8495a32494fe926366e"}
Nov 03 19:47:45.133976 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:45.133832    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r" event={"ID":"8e00f1a1-8156-4fa7-8308-b48127f82aa6","Type":"ContainerStarted","Data":"c80c5955640014d9c6083295d31e98784534ca733a4f90f8331c64acf63db028"}
Nov 03 19:47:45.136658 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:45.136630    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989" event={"ID":"1faa21e4-5bf7-418d-adc8-3fc5bf955010","Type":"ContainerStarted","Data":"382ad4b105b08bb53ffbcbad01efa24da8a59243875b635ae171dba58a67a149"}
Nov 03 19:47:45.136806 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:45.136680    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989" event={"ID":"1faa21e4-5bf7-418d-adc8-3fc5bf955010","Type":"ContainerStarted","Data":"b99c83f5b14b6c9d0d2d24d121db2491e08b97b625d27ebdb8228cc330c610a2"}
Nov 03 19:47:45.141416 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:45.141395    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:47:45.168590 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:45.168533    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r" podStartSLOduration=1.168520217 podStartE2EDuration="1.168520217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:45.16595164 +0000 UTC m=+613.938244660" watchObservedRunningTime="2026-11-03 19:47:45.168520217 +0000 UTC m=+613.940813235"
Nov 03 19:47:45.191313 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:45.190937    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989" podStartSLOduration=2.190921096 podStartE2EDuration="2.190921096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:45.190685158 +0000 UTC m=+613.962978170" watchObservedRunningTime="2026-11-03 19:47:45.190921096 +0000 UTC m=+613.963214113"
Nov 03 19:47:45.288274 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:45.288241    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:45.293282 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:45.293258    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:45.348847 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:45.348817    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d9c6d40-8002-4c1d-a314-b083c06c28c1" path="/var/lib/kubelet/pods/7d9c6d40-8002-4c1d-a314-b083c06c28c1/volumes"
Nov 03 19:47:45.349274 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:45.349259    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3587641-fcd3-488f-b8a9-8e77dc90f934" path="/var/lib/kubelet/pods/a3587641-fcd3-488f-b8a9-8e77dc90f934/volumes"
Nov 03 19:47:45.349588 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:45.349576    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feaac383-b97d-42ad-b021-2e24d707739b" path="/var/lib/kubelet/pods/feaac383-b97d-42ad-b021-2e24d707739b/volumes"
Nov 03 19:47:45.628545 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:45.628508    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:45.814270 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:45.814235    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb"]
Nov 03 19:47:45.814513 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:45.814488    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb" podUID="b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc" containerName="maas-api" containerID="cri-o://de2c2b0ebcfdb902c099ce2f828c450a37e52454bb1a4a9ceddee506113e8dd1" gracePeriod=30
Nov 03 19:47:46.098613 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:46.098579    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb"
Nov 03 19:47:46.160080 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:46.160043    5591 generic.go:356] "Generic (PLEG): container finished" podID="b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc" containerID="de2c2b0ebcfdb902c099ce2f828c450a37e52454bb1a4a9ceddee506113e8dd1" exitCode=0
Nov 03 19:47:46.160291 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:46.160128    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb"
Nov 03 19:47:46.160291 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:46.160134    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb" event={"ID":"b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc","Type":"ContainerDied","Data":"de2c2b0ebcfdb902c099ce2f828c450a37e52454bb1a4a9ceddee506113e8dd1"}
Nov 03 19:47:46.160431 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:46.160288    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb" event={"ID":"b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc","Type":"ContainerDied","Data":"5dc59ec0d0e3f8bf2a61264ac060b0dc636af5f719f246cc38777b24d01e9110"}
Nov 03 19:47:46.160431 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:46.160341    5591 scope.go:122] "RemoveContainer" containerID="de2c2b0ebcfdb902c099ce2f828c450a37e52454bb1a4a9ceddee506113e8dd1"
Nov 03 19:47:46.160813 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:46.160783    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:46.162174 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:46.162143    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:47:46.171468 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:46.171449    5591 scope.go:122] "RemoveContainer" containerID="de2c2b0ebcfdb902c099ce2f828c450a37e52454bb1a4a9ceddee506113e8dd1"
Nov 03 19:47:46.171806 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:47:46.171779    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de2c2b0ebcfdb902c099ce2f828c450a37e52454bb1a4a9ceddee506113e8dd1\": container with ID starting with de2c2b0ebcfdb902c099ce2f828c450a37e52454bb1a4a9ceddee506113e8dd1 not found: ID does not exist" containerID="de2c2b0ebcfdb902c099ce2f828c450a37e52454bb1a4a9ceddee506113e8dd1"
Nov 03 19:47:46.171867 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:46.171817    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de2c2b0ebcfdb902c099ce2f828c450a37e52454bb1a4a9ceddee506113e8dd1"} err="failed to get container status \"de2c2b0ebcfdb902c099ce2f828c450a37e52454bb1a4a9ceddee506113e8dd1\": rpc error: code = NotFound desc = could not find container \"de2c2b0ebcfdb902c099ce2f828c450a37e52454bb1a4a9ceddee506113e8dd1\": container with ID starting with de2c2b0ebcfdb902c099ce2f828c450a37e52454bb1a4a9ceddee506113e8dd1 not found: ID does not exist"
Nov 03 19:47:46.185756 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:46.185686    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-maas-api-tls\") pod \"b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc\" (UID: \"b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc\") "
Nov 03 19:47:46.185858 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:46.185781    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-kube-api-access-ssmlc\" (UniqueName: \"kubernetes.io/projected/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-kube-api-access-ssmlc\") pod \"b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc\" (UID: \"b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc\") "
Nov 03 19:47:46.185858 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:46.185836    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-openshift-service-ca\") pod \"b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc\" (UID: \"b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc\") "
Nov 03 19:47:46.185967 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:46.185862    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-metrics-certs\") pod \"b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc\" (UID: \"b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc\") "
Nov 03 19:47:46.186464 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:46.186436    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-openshift-service-ca" pod "b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc" (UID: "b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:46.188073 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:46.188046    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-maas-api-tls" pod "b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc" (UID: "b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:46.188172 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:46.188114    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-kube-api-access-ssmlc" pod "b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc" (UID: "b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc"). InnerVolumeSpecName "kube-api-access-ssmlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:46.188249 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:46.188232    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-metrics-certs" pod "b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc" (UID: "b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:46.287942 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:46.287723    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:46.287942 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:46.287762    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssmlc\" (UniqueName: \"kubernetes.io/projected/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-kube-api-access-ssmlc\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:46.287942 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:46.287777    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:46.287942 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:46.287791    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:46.484117 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:46.484089    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb"]
Nov 03 19:47:46.489627 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:46.489602    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c20ed791-8876f4c47-q6qwb"]
Nov 03 19:47:46.652636 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:46.652602    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:47.166361 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:47.166333    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:47.167462 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:47.167439    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:47:47.352826 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:47.352784    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc" path="/var/lib/kubelet/pods/b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc/volumes"
Nov 03 19:47:47.377429 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:47.377400    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9"]
Nov 03 19:47:47.378863 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:47.378838    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8c14f244-hcfs9"]
Nov 03 19:47:47.734447 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:47.734414    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"]
Nov 03 19:47:47.734893 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:47.734726    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f" podUID="c7b7ef8f-17e3-4fe9-a98d-cbd45650f626" containerName="istio-proxy" containerID="cri-o://9b593ef82c96e11ceb517d99ed48467d3d3b6dd7ffc592b7b292c2bdb518c6ec" gracePeriod=30
Nov 03 19:47:48.808453 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:48.808414    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7" podUID="5b020e42-25c7-41b6-93fa-ebc2749579a4" containerName="payload-processing" containerID="cri-o://6fc68052e7c5ecb72b8f5134017646d7d02df0c7c54b4d642cc2509c9f28d6d3" gracePeriod=25
Nov 03 19:47:48.832363 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:48.832330    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d" podUID="41174c3b-6b28-4a8c-82e8-acbf40ba454f" containerName="payload-pre-processing" containerID="cri-o://29f78e2cc006f78a5e9659e7a5ccd435eca4db7e5ce7983f655e13b7419be0d6" gracePeriod=25
Nov 03 19:47:48.843805 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:48.843693    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j"]
Nov 03 19:47:48.844446 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:48.844424    5591 cpu_manager.go:414] "containerMap: removing container" podUID="b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc" containerName="maas-api"
Nov 03 19:47:48.844619 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:48.844603    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc" containerName="maas-api"
Nov 03 19:47:48.845084 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:48.844894    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ae0fa5-a3fd-445d-b3e8-f1b71d4e3afc" containerName="maas-api"
Nov 03 19:47:48.848720 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:48.848703    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j"
Nov 03 19:47:48.858624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:48.858597    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:47:48.859533 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:48.859501    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:47:48.861863 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:48.861838    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j"]
Nov 03 19:47:48.879847 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:48.879821    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-9f46e5-serving-cert\""
Nov 03 19:47:48.879973 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:48.879936    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 19:47:48.880051 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:48.880037    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-942q2\""
Nov 03 19:47:48.921435 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:48.921407    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f37dfd-b7be-4333-a7c2-489edff95788-maas-api-tls\") pod \"maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j\" (UID: \"85f37dfd-b7be-4333-a7c2-489edff95788\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j"
Nov 03 19:47:48.921595 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:48.921449    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f37dfd-b7be-4333-a7c2-489edff95788-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j\" (UID: \"85f37dfd-b7be-4333-a7c2-489edff95788\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j"
Nov 03 19:47:48.921595 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:48.921486    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f37dfd-b7be-4333-a7c2-489edff95788-metrics-certs\") pod \"maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j\" (UID: \"85f37dfd-b7be-4333-a7c2-489edff95788\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j"
Nov 03 19:47:48.921595 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:48.921521    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcjq2\" (UniqueName: \"kubernetes.io/projected/85f37dfd-b7be-4333-a7c2-489edff95788-kube-api-access-qcjq2\") pod \"maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j\" (UID: \"85f37dfd-b7be-4333-a7c2-489edff95788\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j"
Nov 03 19:47:49.023725 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.023684    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f37dfd-b7be-4333-a7c2-489edff95788-maas-api-tls\") pod \"maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j\" (UID: \"85f37dfd-b7be-4333-a7c2-489edff95788\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j"
Nov 03 19:47:49.023887 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.023734    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f37dfd-b7be-4333-a7c2-489edff95788-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j\" (UID: \"85f37dfd-b7be-4333-a7c2-489edff95788\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j"
Nov 03 19:47:49.023887 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.023772    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f37dfd-b7be-4333-a7c2-489edff95788-metrics-certs\") pod \"maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j\" (UID: \"85f37dfd-b7be-4333-a7c2-489edff95788\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j"
Nov 03 19:47:49.023887 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.023813    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcjq2\" (UniqueName: \"kubernetes.io/projected/85f37dfd-b7be-4333-a7c2-489edff95788-kube-api-access-qcjq2\") pod \"maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j\" (UID: \"85f37dfd-b7be-4333-a7c2-489edff95788\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j"
Nov 03 19:47:49.025909 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.025854    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f37dfd-b7be-4333-a7c2-489edff95788-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j\" (UID: \"85f37dfd-b7be-4333-a7c2-489edff95788\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j"
Nov 03 19:47:49.031102 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.031073    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f37dfd-b7be-4333-a7c2-489edff95788-maas-api-tls\") pod \"maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j\" (UID: \"85f37dfd-b7be-4333-a7c2-489edff95788\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j"
Nov 03 19:47:49.031860 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.031836    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f37dfd-b7be-4333-a7c2-489edff95788-metrics-certs\") pod \"maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j\" (UID: \"85f37dfd-b7be-4333-a7c2-489edff95788\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j"
Nov 03 19:47:49.046540 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.046509    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcjq2\" (UniqueName: \"kubernetes.io/projected/85f37dfd-b7be-4333-a7c2-489edff95788-kube-api-access-qcjq2\") pod \"maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j\" (UID: \"85f37dfd-b7be-4333-a7c2-489edff95788\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j"
Nov 03 19:47:49.095896 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.095870    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7"
Nov 03 19:47:49.161238 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.161193    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d"
Nov 03 19:47:49.182042 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.181998    5591 generic.go:356] "Generic (PLEG): container finished" podID="5b020e42-25c7-41b6-93fa-ebc2749579a4" containerID="6fc68052e7c5ecb72b8f5134017646d7d02df0c7c54b4d642cc2509c9f28d6d3" exitCode=0
Nov 03 19:47:49.182464 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.182434    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7" event={"ID":"5b020e42-25c7-41b6-93fa-ebc2749579a4","Type":"ContainerDied","Data":"6fc68052e7c5ecb72b8f5134017646d7d02df0c7c54b4d642cc2509c9f28d6d3"}
Nov 03 19:47:49.182606 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.182502    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7" event={"ID":"5b020e42-25c7-41b6-93fa-ebc2749579a4","Type":"ContainerDied","Data":"ac7f3046d9512d4b01c1432d57cd0bd8bb85fca54587e5d80deee6f47e87765e"}
Nov 03 19:47:49.182606 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.182531    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7"
Nov 03 19:47:49.182606 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.182536    5591 scope.go:122] "RemoveContainer" containerID="6fc68052e7c5ecb72b8f5134017646d7d02df0c7c54b4d642cc2509c9f28d6d3"
Nov 03 19:47:49.185422 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.185399    5591 generic.go:356] "Generic (PLEG): container finished" podID="41174c3b-6b28-4a8c-82e8-acbf40ba454f" containerID="29f78e2cc006f78a5e9659e7a5ccd435eca4db7e5ce7983f655e13b7419be0d6" exitCode=0
Nov 03 19:47:49.187368 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.186450    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d"
Nov 03 19:47:49.187368 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.186578    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d" event={"ID":"41174c3b-6b28-4a8c-82e8-acbf40ba454f","Type":"ContainerDied","Data":"29f78e2cc006f78a5e9659e7a5ccd435eca4db7e5ce7983f655e13b7419be0d6"}
Nov 03 19:47:49.187368 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.186617    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d" event={"ID":"41174c3b-6b28-4a8c-82e8-acbf40ba454f","Type":"ContainerDied","Data":"d6c5a8e171a37903112deb01de937685f48e39e6954d298fba329bc88125c8cc"}
Nov 03 19:47:49.212579 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.208680    5591 scope.go:122] "RemoveContainer" containerID="6fc68052e7c5ecb72b8f5134017646d7d02df0c7c54b4d642cc2509c9f28d6d3"
Nov 03 19:47:49.212579 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:47:49.209843    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fc68052e7c5ecb72b8f5134017646d7d02df0c7c54b4d642cc2509c9f28d6d3\": container with ID starting with 6fc68052e7c5ecb72b8f5134017646d7d02df0c7c54b4d642cc2509c9f28d6d3 not found: ID does not exist" containerID="6fc68052e7c5ecb72b8f5134017646d7d02df0c7c54b4d642cc2509c9f28d6d3"
Nov 03 19:47:49.212579 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.209882    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fc68052e7c5ecb72b8f5134017646d7d02df0c7c54b4d642cc2509c9f28d6d3"} err="failed to get container status \"6fc68052e7c5ecb72b8f5134017646d7d02df0c7c54b4d642cc2509c9f28d6d3\": rpc error: code = NotFound desc = could not find container \"6fc68052e7c5ecb72b8f5134017646d7d02df0c7c54b4d642cc2509c9f28d6d3\": container with ID starting with 6fc68052e7c5ecb72b8f5134017646d7d02df0c7c54b4d642cc2509c9f28d6d3 not found: ID does not exist"
Nov 03 19:47:49.212579 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.209908    5591 scope.go:122] "RemoveContainer" containerID="29f78e2cc006f78a5e9659e7a5ccd435eca4db7e5ce7983f655e13b7419be0d6"
Nov 03 19:47:49.225060 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.225036    5591 scope.go:122] "RemoveContainer" containerID="29f78e2cc006f78a5e9659e7a5ccd435eca4db7e5ce7983f655e13b7419be0d6"
Nov 03 19:47:49.225438 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:47:49.225419    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f78e2cc006f78a5e9659e7a5ccd435eca4db7e5ce7983f655e13b7419be0d6\": container with ID starting with 29f78e2cc006f78a5e9659e7a5ccd435eca4db7e5ce7983f655e13b7419be0d6 not found: ID does not exist" containerID="29f78e2cc006f78a5e9659e7a5ccd435eca4db7e5ce7983f655e13b7419be0d6"
Nov 03 19:47:49.225552 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.225446    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f78e2cc006f78a5e9659e7a5ccd435eca4db7e5ce7983f655e13b7419be0d6"} err="failed to get container status \"29f78e2cc006f78a5e9659e7a5ccd435eca4db7e5ce7983f655e13b7419be0d6\": rpc error: code = NotFound desc = could not find container \"29f78e2cc006f78a5e9659e7a5ccd435eca4db7e5ce7983f655e13b7419be0d6\": container with ID starting with 29f78e2cc006f78a5e9659e7a5ccd435eca4db7e5ce7983f655e13b7419be0d6 not found: ID does not exist"
Nov 03 19:47:49.225597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.225557    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b020e42-25c7-41b6-93fa-ebc2749579a4-kube-api-access-ct7r4\" (UniqueName: \"kubernetes.io/projected/5b020e42-25c7-41b6-93fa-ebc2749579a4-kube-api-access-ct7r4\") pod \"5b020e42-25c7-41b6-93fa-ebc2749579a4\" (UID: \"5b020e42-25c7-41b6-93fa-ebc2749579a4\") "
Nov 03 19:47:49.225660 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.225641    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41174c3b-6b28-4a8c-82e8-acbf40ba454f-kube-api-access-jmtrd\" (UniqueName: \"kubernetes.io/projected/41174c3b-6b28-4a8c-82e8-acbf40ba454f-kube-api-access-jmtrd\") pod \"41174c3b-6b28-4a8c-82e8-acbf40ba454f\" (UID: \"41174c3b-6b28-4a8c-82e8-acbf40ba454f\") "
Nov 03 19:47:49.225782 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.225732    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b020e42-25c7-41b6-93fa-ebc2749579a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b020e42-25c7-41b6-93fa-ebc2749579a4-plugins-config-volume\") pod \"5b020e42-25c7-41b6-93fa-ebc2749579a4\" (UID: \"5b020e42-25c7-41b6-93fa-ebc2749579a4\") "
Nov 03 19:47:49.225866 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.225850    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41174c3b-6b28-4a8c-82e8-acbf40ba454f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41174c3b-6b28-4a8c-82e8-acbf40ba454f-plugins-config-volume\") pod \"41174c3b-6b28-4a8c-82e8-acbf40ba454f\" (UID: \"41174c3b-6b28-4a8c-82e8-acbf40ba454f\") "
Nov 03 19:47:49.226315 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.226091    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b020e42-25c7-41b6-93fa-ebc2749579a4-plugins-config-volume" pod "5b020e42-25c7-41b6-93fa-ebc2749579a4" (UID: "5b020e42-25c7-41b6-93fa-ebc2749579a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:49.226315 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.226272    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41174c3b-6b28-4a8c-82e8-acbf40ba454f-plugins-config-volume" pod "41174c3b-6b28-4a8c-82e8-acbf40ba454f" (UID: "41174c3b-6b28-4a8c-82e8-acbf40ba454f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:49.226315 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.226289    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b020e42-25c7-41b6-93fa-ebc2749579a4-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:49.227924 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.227888    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41174c3b-6b28-4a8c-82e8-acbf40ba454f-kube-api-access-jmtrd" pod "41174c3b-6b28-4a8c-82e8-acbf40ba454f" (UID: "41174c3b-6b28-4a8c-82e8-acbf40ba454f"). InnerVolumeSpecName "kube-api-access-jmtrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:49.228025 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.227970    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b020e42-25c7-41b6-93fa-ebc2749579a4-kube-api-access-ct7r4" pod "5b020e42-25c7-41b6-93fa-ebc2749579a4" (UID: "5b020e42-25c7-41b6-93fa-ebc2749579a4"). InnerVolumeSpecName "kube-api-access-ct7r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:49.235248 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.235225    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j"
Nov 03 19:47:49.328799 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.328709    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41174c3b-6b28-4a8c-82e8-acbf40ba454f-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:49.328799 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.328743    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct7r4\" (UniqueName: \"kubernetes.io/projected/5b020e42-25c7-41b6-93fa-ebc2749579a4-kube-api-access-ct7r4\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:49.328799 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.328760    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmtrd\" (UniqueName: \"kubernetes.io/projected/41174c3b-6b28-4a8c-82e8-acbf40ba454f-kube-api-access-jmtrd\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:49.348296 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.348263    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab1a8953-14f8-4983-a423-fa416f9ec7ae" path="/var/lib/kubelet/pods/ab1a8953-14f8-4983-a423-fa416f9ec7ae/volumes"
Nov 03 19:47:49.503469 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.503438    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7"]
Nov 03 19:47:49.513483 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.513454    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c20ed791-b89f7d964-5bsl7"]
Nov 03 19:47:49.524979 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.524946    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d"]
Nov 03 19:47:49.534653 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.534628    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j"]
Nov 03 19:47:49.535344 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.535321    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c20ed791-649568bc75-gnp7d"]
Nov 03 19:47:49.536832 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:47:49.536808    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f37dfd_b7be_4333_a7c2_489edff95788.slice/crio-4b8aab70f1b8e84627a7bd16b6b24527c73f4b5c40c13c21f4d99a1ae95aa086 WatchSource:0}: Error finding container 4b8aab70f1b8e84627a7bd16b6b24527c73f4b5c40c13c21f4d99a1ae95aa086: Status 404 returned error can't find the container with id 4b8aab70f1b8e84627a7bd16b6b24527c73f4b5c40c13c21f4d99a1ae95aa086
Nov 03 19:47:49.732354 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.732278    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh"]
Nov 03 19:47:49.732758 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.732744    5591 cpu_manager.go:414] "containerMap: removing container" podUID="5b020e42-25c7-41b6-93fa-ebc2749579a4" containerName="payload-processing"
Nov 03 19:47:49.732831 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.732760    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b020e42-25c7-41b6-93fa-ebc2749579a4" containerName="payload-processing"
Nov 03 19:47:49.732831 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.732787    5591 cpu_manager.go:414] "containerMap: removing container" podUID="41174c3b-6b28-4a8c-82e8-acbf40ba454f" containerName="payload-pre-processing"
Nov 03 19:47:49.732831 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.732792    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41174c3b-6b28-4a8c-82e8-acbf40ba454f" containerName="payload-pre-processing"
Nov 03 19:47:49.732940 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.732873    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b020e42-25c7-41b6-93fa-ebc2749579a4" containerName="payload-processing"
Nov 03 19:47:49.732940 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.732883    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="41174c3b-6b28-4a8c-82e8-acbf40ba454f" containerName="payload-pre-processing"
Nov 03 19:47:49.738034 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.738006    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh"
Nov 03 19:47:49.748487 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.744982    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-2b5703\""
Nov 03 19:47:49.748487 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.747744    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh"]
Nov 03 19:47:49.748487 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.747886    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-2b5703-dockercfg-b84j2\""
Nov 03 19:47:49.846353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.846239    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwkqd\" (UniqueName: \"kubernetes.io/projected/190ad217-4f23-4513-be1b-405c141d460d-kube-api-access-nwkqd\") pod \"payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh\" (UID: \"190ad217-4f23-4513-be1b-405c141d460d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh"
Nov 03 19:47:49.846792 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.846361    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/190ad217-4f23-4513-be1b-405c141d460d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh\" (UID: \"190ad217-4f23-4513-be1b-405c141d460d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh"
Nov 03 19:47:49.949636 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.949585    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/190ad217-4f23-4513-be1b-405c141d460d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh\" (UID: \"190ad217-4f23-4513-be1b-405c141d460d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh"
Nov 03 19:47:49.949830 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.949736    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwkqd\" (UniqueName: \"kubernetes.io/projected/190ad217-4f23-4513-be1b-405c141d460d-kube-api-access-nwkqd\") pod \"payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh\" (UID: \"190ad217-4f23-4513-be1b-405c141d460d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh"
Nov 03 19:47:49.950311 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.950287    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/190ad217-4f23-4513-be1b-405c141d460d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh\" (UID: \"190ad217-4f23-4513-be1b-405c141d460d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh"
Nov 03 19:47:49.962520 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:49.962492    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwkqd\" (UniqueName: \"kubernetes.io/projected/190ad217-4f23-4513-be1b-405c141d460d-kube-api-access-nwkqd\") pod \"payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh\" (UID: \"190ad217-4f23-4513-be1b-405c141d460d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh"
Nov 03 19:47:50.055596 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.055569    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh"
Nov 03 19:47:50.078460 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.078434    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:50.152947 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.152595    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27e0a187-19d5-45cb-bdd6-711f30368cca-kube-api-access-bm9hs\" (UniqueName: \"kubernetes.io/projected/27e0a187-19d5-45cb-bdd6-711f30368cca-kube-api-access-bm9hs\") pod \"27e0a187-19d5-45cb-bdd6-711f30368cca\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") "
Nov 03 19:47:50.152947 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.152631    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-token\" (UniqueName: \"kubernetes.io/projected/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-token\") pod \"27e0a187-19d5-45cb-bdd6-711f30368cca\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") "
Nov 03 19:47:50.152947 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.152653    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-credential-socket\") pod \"27e0a187-19d5-45cb-bdd6-711f30368cca\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") "
Nov 03 19:47:50.152947 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.152799    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-workload-certs\") pod \"27e0a187-19d5-45cb-bdd6-711f30368cca\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") "
Nov 03 19:47:50.152947 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.152831    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-podinfo\") pod \"27e0a187-19d5-45cb-bdd6-711f30368cca\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") "
Nov 03 19:47:50.152947 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.152855    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27e0a187-19d5-45cb-bdd6-711f30368cca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27e0a187-19d5-45cb-bdd6-711f30368cca-istiod-ca-cert\") pod \"27e0a187-19d5-45cb-bdd6-711f30368cca\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") "
Nov 03 19:47:50.152947 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.152874    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-workload-socket\") pod \"27e0a187-19d5-45cb-bdd6-711f30368cca\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") "
Nov 03 19:47:50.152947 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.152891    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-envoy\") pod \"27e0a187-19d5-45cb-bdd6-711f30368cca\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") "
Nov 03 19:47:50.152947 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.152937    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-data\") pod \"27e0a187-19d5-45cb-bdd6-711f30368cca\" (UID: \"27e0a187-19d5-45cb-bdd6-711f30368cca\") "
Nov 03 19:47:50.156615 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.153443    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-data" pod "27e0a187-19d5-45cb-bdd6-711f30368cca" (UID: "27e0a187-19d5-45cb-bdd6-711f30368cca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:47:50.156615 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.154643    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-credential-socket" pod "27e0a187-19d5-45cb-bdd6-711f30368cca" (UID: "27e0a187-19d5-45cb-bdd6-711f30368cca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:47:50.156615 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.154728    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-workload-certs" pod "27e0a187-19d5-45cb-bdd6-711f30368cca" (UID: "27e0a187-19d5-45cb-bdd6-711f30368cca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:47:50.156615 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.154878    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27e0a187-19d5-45cb-bdd6-711f30368cca-istiod-ca-cert" pod "27e0a187-19d5-45cb-bdd6-711f30368cca" (UID: "27e0a187-19d5-45cb-bdd6-711f30368cca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:50.156615 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.154927    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-workload-socket" pod "27e0a187-19d5-45cb-bdd6-711f30368cca" (UID: "27e0a187-19d5-45cb-bdd6-711f30368cca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:47:50.157194 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.157077    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e0a187-19d5-45cb-bdd6-711f30368cca-kube-api-access-bm9hs" pod "27e0a187-19d5-45cb-bdd6-711f30368cca" (UID: "27e0a187-19d5-45cb-bdd6-711f30368cca"). InnerVolumeSpecName "kube-api-access-bm9hs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:50.162270 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.158646    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-token" pod "27e0a187-19d5-45cb-bdd6-711f30368cca" (UID: "27e0a187-19d5-45cb-bdd6-711f30368cca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:50.162270 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.158721    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-podinfo" pod "27e0a187-19d5-45cb-bdd6-711f30368cca" (UID: "27e0a187-19d5-45cb-bdd6-711f30368cca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:47:50.162270 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.158809    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-envoy" pod "27e0a187-19d5-45cb-bdd6-711f30368cca" (UID: "27e0a187-19d5-45cb-bdd6-711f30368cca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:47:50.195669 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.195632    5591 generic.go:356] "Generic (PLEG): container finished" podID="27e0a187-19d5-45cb-bdd6-711f30368cca" containerID="0564a244161a135c39f6148d98cb1c83d1fe40765424aa0286014d3b86be34b2" exitCode=0
Nov 03 19:47:50.195839 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.195785    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"
Nov 03 19:47:50.196538 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.196502    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr" event={"ID":"27e0a187-19d5-45cb-bdd6-711f30368cca","Type":"ContainerDied","Data":"0564a244161a135c39f6148d98cb1c83d1fe40765424aa0286014d3b86be34b2"}
Nov 03 19:47:50.196660 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.196574    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr" event={"ID":"27e0a187-19d5-45cb-bdd6-711f30368cca","Type":"ContainerDied","Data":"45304c163d644d5fcdf4a04d2becc0c1c9f59f9ea088e2ca0ad06eb16a174d36"}
Nov 03 19:47:50.196660 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.196605    5591 scope.go:122] "RemoveContainer" containerID="0564a244161a135c39f6148d98cb1c83d1fe40765424aa0286014d3b86be34b2"
Nov 03 19:47:50.202798 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.202654    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j" event={"ID":"85f37dfd-b7be-4333-a7c2-489edff95788","Type":"ContainerStarted","Data":"d8f914ec65b8ca458c7f4cc3b39830cdd403596c79e780bd4b3ac95489ec7cbb"}
Nov 03 19:47:50.202798 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.202715    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j"
Nov 03 19:47:50.202798 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.202760    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j" event={"ID":"85f37dfd-b7be-4333-a7c2-489edff95788","Type":"ContainerStarted","Data":"4b8aab70f1b8e84627a7bd16b6b24527c73f4b5c40c13c21f4d99a1ae95aa086"}
Nov 03 19:47:50.207528 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.207512    5591 scope.go:122] "RemoveContainer" containerID="f21e61b950bb814337c21e1a66b772f692528552543377abd1677bf686013934"
Nov 03 19:47:50.215859 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.215844    5591 scope.go:122] "RemoveContainer" containerID="0564a244161a135c39f6148d98cb1c83d1fe40765424aa0286014d3b86be34b2"
Nov 03 19:47:50.216120 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:47:50.216101    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0564a244161a135c39f6148d98cb1c83d1fe40765424aa0286014d3b86be34b2\": container with ID starting with 0564a244161a135c39f6148d98cb1c83d1fe40765424aa0286014d3b86be34b2 not found: ID does not exist" containerID="0564a244161a135c39f6148d98cb1c83d1fe40765424aa0286014d3b86be34b2"
Nov 03 19:47:50.216183 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.216129    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0564a244161a135c39f6148d98cb1c83d1fe40765424aa0286014d3b86be34b2"} err="failed to get container status \"0564a244161a135c39f6148d98cb1c83d1fe40765424aa0286014d3b86be34b2\": rpc error: code = NotFound desc = could not find container \"0564a244161a135c39f6148d98cb1c83d1fe40765424aa0286014d3b86be34b2\": container with ID starting with 0564a244161a135c39f6148d98cb1c83d1fe40765424aa0286014d3b86be34b2 not found: ID does not exist"
Nov 03 19:47:50.216183 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.216146    5591 scope.go:122] "RemoveContainer" containerID="f21e61b950bb814337c21e1a66b772f692528552543377abd1677bf686013934"
Nov 03 19:47:50.216402 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:47:50.216384    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f21e61b950bb814337c21e1a66b772f692528552543377abd1677bf686013934\": container with ID starting with f21e61b950bb814337c21e1a66b772f692528552543377abd1677bf686013934 not found: ID does not exist" containerID="f21e61b950bb814337c21e1a66b772f692528552543377abd1677bf686013934"
Nov 03 19:47:50.216443 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.216412    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f21e61b950bb814337c21e1a66b772f692528552543377abd1677bf686013934"} err="failed to get container status \"f21e61b950bb814337c21e1a66b772f692528552543377abd1677bf686013934\": rpc error: code = NotFound desc = could not find container \"f21e61b950bb814337c21e1a66b772f692528552543377abd1677bf686013934\": container with ID starting with f21e61b950bb814337c21e1a66b772f692528552543377abd1677bf686013934 not found: ID does not exist"
Nov 03 19:47:50.223398 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.223357    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j" podStartSLOduration=1.7938669329999999 podStartE2EDuration="2.22334372s" totalImagesPullingTime="429.476787ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:50.221988141 +0000 UTC m=+618.994281160" watchObservedRunningTime="2026-11-03 19:47:50.22334372 +0000 UTC m=+618.995636734"
Nov 03 19:47:50.237812 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.237760    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"]
Nov 03 19:47:50.241898 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.241879    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3f7e10b8-openshift-default-78c6f9c4f7-zgtvr"]
Nov 03 19:47:50.254341 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.254306    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:50.254341 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.254345    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:50.254480 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.254362    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27e0a187-19d5-45cb-bdd6-711f30368cca-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:50.254480 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.254378    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:50.254480 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.254394    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:50.254480 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.254408    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:50.254480 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.254422    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm9hs\" (UniqueName: \"kubernetes.io/projected/27e0a187-19d5-45cb-bdd6-711f30368cca-kube-api-access-bm9hs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:50.254480 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.254438    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27e0a187-19d5-45cb-bdd6-711f30368cca-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:50.254480 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.254454    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27e0a187-19d5-45cb-bdd6-711f30368cca-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:50.387623 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:50.386277    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh"]
Nov 03 19:47:50.389518 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:47:50.389489    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190ad217_4f23_4513_be1b_405c141d460d.slice/crio-86aa1dc3b379ca4fb8046ebb7f4b173a0d4929622b3496544e0e25240705b8c9 WatchSource:0}: Error finding container 86aa1dc3b379ca4fb8046ebb7f4b173a0d4929622b3496544e0e25240705b8c9: Status 404 returned error can't find the container with id 86aa1dc3b379ca4fb8046ebb7f4b173a0d4929622b3496544e0e25240705b8c9
Nov 03 19:47:51.210851 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.210762    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh" event={"ID":"190ad217-4f23-4513-be1b-405c141d460d","Type":"ContainerStarted","Data":"5e456c3438869f653ffb43d9ff20f40e1e1d07b0db03378218c9484b16f789a0"}
Nov 03 19:47:51.210851 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.210825    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh" event={"ID":"190ad217-4f23-4513-be1b-405c141d460d","Type":"ContainerStarted","Data":"86aa1dc3b379ca4fb8046ebb7f4b173a0d4929622b3496544e0e25240705b8c9"}
Nov 03 19:47:51.211353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.210969    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh"
Nov 03 19:47:51.229619 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.229565    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh" podStartSLOduration=2.229549014 podStartE2EDuration="2.229549014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:51.228440064 +0000 UTC m=+620.000733093" watchObservedRunningTime="2026-11-03 19:47:51.229549014 +0000 UTC m=+620.001842031"
Nov 03 19:47:51.353917 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.350384    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27e0a187-19d5-45cb-bdd6-711f30368cca" path="/var/lib/kubelet/pods/27e0a187-19d5-45cb-bdd6-711f30368cca/volumes"
Nov 03 19:47:51.353917 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.350957    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41174c3b-6b28-4a8c-82e8-acbf40ba454f" path="/var/lib/kubelet/pods/41174c3b-6b28-4a8c-82e8-acbf40ba454f/volumes"
Nov 03 19:47:51.354537 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.354511    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b020e42-25c7-41b6-93fa-ebc2749579a4" path="/var/lib/kubelet/pods/5b020e42-25c7-41b6-93fa-ebc2749579a4/volumes"
Nov 03 19:47:51.684658 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.684620    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb"]
Nov 03 19:47:51.685245 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.685219    5591 cpu_manager.go:414] "containerMap: removing container" podUID="27e0a187-19d5-45cb-bdd6-711f30368cca" containerName="istio-proxy"
Nov 03 19:47:51.685245 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.685244    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27e0a187-19d5-45cb-bdd6-711f30368cca" containerName="istio-proxy"
Nov 03 19:47:51.685421 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.685388    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="27e0a187-19d5-45cb-bdd6-711f30368cca" containerName="istio-proxy"
Nov 03 19:47:51.689874 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.689857    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb"
Nov 03 19:47:51.700509 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.700480    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-d3d0df\""
Nov 03 19:47:51.703335 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.703305    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb"]
Nov 03 19:47:51.725229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.725164    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm"]
Nov 03 19:47:51.725959 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.725934    5591 cpu_manager.go:414] "containerMap: removing container" podUID="27e0a187-19d5-45cb-bdd6-711f30368cca" containerName="istio-proxy"
Nov 03 19:47:51.726142 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.726114    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27e0a187-19d5-45cb-bdd6-711f30368cca" containerName="istio-proxy"
Nov 03 19:47:51.726293 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.726276    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="27e0a187-19d5-45cb-bdd6-711f30368cca" containerName="istio-proxy"
Nov 03 19:47:51.730084 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.730061    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm"
Nov 03 19:47:51.735906 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.735885    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-d3d0df-dockercfg-dchgx\""
Nov 03 19:47:51.740170 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.740147    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm"]
Nov 03 19:47:51.771624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.771570    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed913799-a594-487c-92a1-f55b50cac03c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb\" (UID: \"ed913799-a594-487c-92a1-f55b50cac03c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb"
Nov 03 19:47:51.771811 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.771654    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a74714b9-b503-4817-b5d2-9ab74cd798a7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm\" (UID: \"a74714b9-b503-4817-b5d2-9ab74cd798a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm"
Nov 03 19:47:51.771811 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.771748    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfhdg\" (UniqueName: \"kubernetes.io/projected/a74714b9-b503-4817-b5d2-9ab74cd798a7-kube-api-access-lfhdg\") pod \"payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm\" (UID: \"a74714b9-b503-4817-b5d2-9ab74cd798a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm"
Nov 03 19:47:51.771966 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.771810    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp26f\" (UniqueName: \"kubernetes.io/projected/ed913799-a594-487c-92a1-f55b50cac03c-kube-api-access-sp26f\") pod \"payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb\" (UID: \"ed913799-a594-487c-92a1-f55b50cac03c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb"
Nov 03 19:47:51.872774 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.872728    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed913799-a594-487c-92a1-f55b50cac03c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb\" (UID: \"ed913799-a594-487c-92a1-f55b50cac03c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb"
Nov 03 19:47:51.872938 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.872824    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a74714b9-b503-4817-b5d2-9ab74cd798a7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm\" (UID: \"a74714b9-b503-4817-b5d2-9ab74cd798a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm"
Nov 03 19:47:51.872938 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.872864    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfhdg\" (UniqueName: \"kubernetes.io/projected/a74714b9-b503-4817-b5d2-9ab74cd798a7-kube-api-access-lfhdg\") pod \"payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm\" (UID: \"a74714b9-b503-4817-b5d2-9ab74cd798a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm"
Nov 03 19:47:51.872938 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.872915    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp26f\" (UniqueName: \"kubernetes.io/projected/ed913799-a594-487c-92a1-f55b50cac03c-kube-api-access-sp26f\") pod \"payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb\" (UID: \"ed913799-a594-487c-92a1-f55b50cac03c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb"
Nov 03 19:47:51.873523 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.873500    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed913799-a594-487c-92a1-f55b50cac03c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb\" (UID: \"ed913799-a594-487c-92a1-f55b50cac03c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb"
Nov 03 19:47:51.873523 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.873517    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a74714b9-b503-4817-b5d2-9ab74cd798a7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm\" (UID: \"a74714b9-b503-4817-b5d2-9ab74cd798a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm"
Nov 03 19:47:51.885429 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.885401    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfhdg\" (UniqueName: \"kubernetes.io/projected/a74714b9-b503-4817-b5d2-9ab74cd798a7-kube-api-access-lfhdg\") pod \"payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm\" (UID: \"a74714b9-b503-4817-b5d2-9ab74cd798a7\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm"
Nov 03 19:47:51.885577 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:51.885403    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp26f\" (UniqueName: \"kubernetes.io/projected/ed913799-a594-487c-92a1-f55b50cac03c-kube-api-access-sp26f\") pod \"payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb\" (UID: \"ed913799-a594-487c-92a1-f55b50cac03c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb"
Nov 03 19:47:52.012284 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:52.012174    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb"
Nov 03 19:47:52.049826 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:52.049788    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm"
Nov 03 19:47:52.205501 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:52.205468    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:47:52.302904 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:52.302873    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb"]
Nov 03 19:47:52.305238 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:47:52.305183    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded913799_a594_487c_92a1_f55b50cac03c.slice/crio-7bffa589610413c4f2085f48c24761ce7fa956abe09d0925ad424b199ced7cca WatchSource:0}: Error finding container 7bffa589610413c4f2085f48c24761ce7fa956abe09d0925ad424b199ced7cca: Status 404 returned error can't find the container with id 7bffa589610413c4f2085f48c24761ce7fa956abe09d0925ad424b199ced7cca
Nov 03 19:47:52.352080 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:52.352056    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm"]
Nov 03 19:47:52.354136 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:47:52.354105    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda74714b9_b503_4817_b5d2_9ab74cd798a7.slice/crio-c5d45575c48ef1cbb626ba17f44ba0fff510d25d9cea039f3061421aad205d7d WatchSource:0}: Error finding container c5d45575c48ef1cbb626ba17f44ba0fff510d25d9cea039f3061421aad205d7d: Status 404 returned error can't find the container with id c5d45575c48ef1cbb626ba17f44ba0fff510d25d9cea039f3061421aad205d7d
Nov 03 19:47:52.429323 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:52.429291    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4"]
Nov 03 19:47:52.433934 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:52.433907    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4"
Nov 03 19:47:52.443473 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:52.443448    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4"]
Nov 03 19:47:52.447043 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:52.447011    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-d3d0df-serving-cert\""
Nov 03 19:47:52.497999 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:52.497966    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de96244c-7272-46a0-b20b-5cd1470fc8bc-metrics-certs\") pod \"maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4\" (UID: \"de96244c-7272-46a0-b20b-5cd1470fc8bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4"
Nov 03 19:47:52.498186 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:52.498119    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtrm5\" (UniqueName: \"kubernetes.io/projected/de96244c-7272-46a0-b20b-5cd1470fc8bc-kube-api-access-jtrm5\") pod \"maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4\" (UID: \"de96244c-7272-46a0-b20b-5cd1470fc8bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4"
Nov 03 19:47:52.498186 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:52.498156    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de96244c-7272-46a0-b20b-5cd1470fc8bc-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4\" (UID: \"de96244c-7272-46a0-b20b-5cd1470fc8bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4"
Nov 03 19:47:52.498352 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:52.498270    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de96244c-7272-46a0-b20b-5cd1470fc8bc-maas-api-tls\") pod \"maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4\" (UID: \"de96244c-7272-46a0-b20b-5cd1470fc8bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4"
Nov 03 19:47:52.599804 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:52.599714    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de96244c-7272-46a0-b20b-5cd1470fc8bc-metrics-certs\") pod \"maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4\" (UID: \"de96244c-7272-46a0-b20b-5cd1470fc8bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4"
Nov 03 19:47:52.600869 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:52.600839    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtrm5\" (UniqueName: \"kubernetes.io/projected/de96244c-7272-46a0-b20b-5cd1470fc8bc-kube-api-access-jtrm5\") pod \"maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4\" (UID: \"de96244c-7272-46a0-b20b-5cd1470fc8bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4"
Nov 03 19:47:52.601463 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:52.601442    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de96244c-7272-46a0-b20b-5cd1470fc8bc-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4\" (UID: \"de96244c-7272-46a0-b20b-5cd1470fc8bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4"
Nov 03 19:47:52.623270 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:52.623243    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de96244c-7272-46a0-b20b-5cd1470fc8bc-maas-api-tls\") pod \"maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4\" (UID: \"de96244c-7272-46a0-b20b-5cd1470fc8bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4"
Nov 03 19:47:52.623557 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:47:52.623529    5591 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-serving-cert: secret "maas-api-e2e-worker-w0-d3d0df-serving-cert" not found
Nov 03 19:47:52.623719 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:47:52.623611    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de96244c-7272-46a0-b20b-5cd1470fc8bc-maas-api-tls podName:de96244c-7272-46a0-b20b-5cd1470fc8bc nodeName:}" failed. No retries permitted until 2026-11-03 19:47:53.12358668 +0000 UTC m=+621.895879699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/de96244c-7272-46a0-b20b-5cd1470fc8bc-maas-api-tls") pod "maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4" (UID: "de96244c-7272-46a0-b20b-5cd1470fc8bc") : secret "maas-api-e2e-worker-w0-d3d0df-serving-cert" not found
Nov 03 19:47:52.623719 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:52.621911    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de96244c-7272-46a0-b20b-5cd1470fc8bc-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4\" (UID: \"de96244c-7272-46a0-b20b-5cd1470fc8bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4"
Nov 03 19:47:52.623719 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:52.618875    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de96244c-7272-46a0-b20b-5cd1470fc8bc-metrics-certs\") pod \"maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4\" (UID: \"de96244c-7272-46a0-b20b-5cd1470fc8bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4"
Nov 03 19:47:52.624178 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:52.624148    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtrm5\" (UniqueName: \"kubernetes.io/projected/de96244c-7272-46a0-b20b-5cd1470fc8bc-kube-api-access-jtrm5\") pod \"maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4\" (UID: \"de96244c-7272-46a0-b20b-5cd1470fc8bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4"
Nov 03 19:47:53.022025 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.021990    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:53.128787 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.128674    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-data\") pod \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") "
Nov 03 19:47:53.128787 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.128752    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-credential-socket\") pod \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") "
Nov 03 19:47:53.129017 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.128791    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-envoy\") pod \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") "
Nov 03 19:47:53.129017 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.128848    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-workload-socket\") pod \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") "
Nov 03 19:47:53.129017 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.128878    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-kube-api-access-jwn9t\" (UniqueName: \"kubernetes.io/projected/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-kube-api-access-jwn9t\") pod \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") "
Nov 03 19:47:53.129017 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.128916    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istiod-ca-cert\") pod \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") "
Nov 03 19:47:53.129017 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.128981    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-data" pod "c7b7ef8f-17e3-4fe9-a98d-cbd45650f626" (UID: "c7b7ef8f-17e3-4fe9-a98d-cbd45650f626"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:47:53.129318 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.129096    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-workload-certs\") pod \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") "
Nov 03 19:47:53.129318 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.129158    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-workload-socket" pod "c7b7ef8f-17e3-4fe9-a98d-cbd45650f626" (UID: "c7b7ef8f-17e3-4fe9-a98d-cbd45650f626"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:47:53.129318 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.129177    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-credential-socket" pod "c7b7ef8f-17e3-4fe9-a98d-cbd45650f626" (UID: "c7b7ef8f-17e3-4fe9-a98d-cbd45650f626"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:47:53.129318 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.129197    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-token\" (UniqueName: \"kubernetes.io/projected/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-token\") pod \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") "
Nov 03 19:47:53.129516 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.129453    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-podinfo\") pod \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\" (UID: \"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626\") "
Nov 03 19:47:53.129801 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.129748    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-workload-certs" pod "c7b7ef8f-17e3-4fe9-a98d-cbd45650f626" (UID: "c7b7ef8f-17e3-4fe9-a98d-cbd45650f626"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:47:53.129966 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.129847    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istiod-ca-cert" pod "c7b7ef8f-17e3-4fe9-a98d-cbd45650f626" (UID: "c7b7ef8f-17e3-4fe9-a98d-cbd45650f626"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:53.129966 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.129891    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de96244c-7272-46a0-b20b-5cd1470fc8bc-maas-api-tls\") pod \"maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4\" (UID: \"de96244c-7272-46a0-b20b-5cd1470fc8bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4"
Nov 03 19:47:53.130106 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.130006    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:53.130106 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.130023    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:53.130106 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.130038    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:53.130106 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.130051    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:53.130106 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.130066    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:53.132052 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.132016    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-kube-api-access-jwn9t" pod "c7b7ef8f-17e3-4fe9-a98d-cbd45650f626" (UID: "c7b7ef8f-17e3-4fe9-a98d-cbd45650f626"). InnerVolumeSpecName "kube-api-access-jwn9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:53.132195 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.132177    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-envoy" pod "c7b7ef8f-17e3-4fe9-a98d-cbd45650f626" (UID: "c7b7ef8f-17e3-4fe9-a98d-cbd45650f626"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:47:53.132418 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.132390    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-token" pod "c7b7ef8f-17e3-4fe9-a98d-cbd45650f626" (UID: "c7b7ef8f-17e3-4fe9-a98d-cbd45650f626"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:53.132743 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.132722    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-podinfo" pod "c7b7ef8f-17e3-4fe9-a98d-cbd45650f626" (UID: "c7b7ef8f-17e3-4fe9-a98d-cbd45650f626"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:47:53.133035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.133014    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de96244c-7272-46a0-b20b-5cd1470fc8bc-maas-api-tls\") pod \"maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4\" (UID: \"de96244c-7272-46a0-b20b-5cd1470fc8bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4"
Nov 03 19:47:53.224052 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.224021    5591 generic.go:356] "Generic (PLEG): container finished" podID="c7b7ef8f-17e3-4fe9-a98d-cbd45650f626" containerID="9b593ef82c96e11ceb517d99ed48467d3d3b6dd7ffc592b7b292c2bdb518c6ec" exitCode=0
Nov 03 19:47:53.224454 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.224075    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"
Nov 03 19:47:53.224454 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.224108    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f" event={"ID":"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626","Type":"ContainerDied","Data":"9b593ef82c96e11ceb517d99ed48467d3d3b6dd7ffc592b7b292c2bdb518c6ec"}
Nov 03 19:47:53.224454 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.224162    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f" event={"ID":"c7b7ef8f-17e3-4fe9-a98d-cbd45650f626","Type":"ContainerDied","Data":"7e020a100ebd15cb30c8222685dd7f07b9fe8815c46cfe94d48b570e58284dba"}
Nov 03 19:47:53.224454 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.224185    5591 scope.go:122] "RemoveContainer" containerID="9b593ef82c96e11ceb517d99ed48467d3d3b6dd7ffc592b7b292c2bdb518c6ec"
Nov 03 19:47:53.226052 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.226027    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb" event={"ID":"ed913799-a594-487c-92a1-f55b50cac03c","Type":"ContainerStarted","Data":"ee7bc8ed15cc181aa29bef086cc5ae20e13c64754b64b2c6da1dc86594f63733"}
Nov 03 19:47:53.226160 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.226089    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb"
Nov 03 19:47:53.226160 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.226114    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb" event={"ID":"ed913799-a594-487c-92a1-f55b50cac03c","Type":"ContainerStarted","Data":"7bffa589610413c4f2085f48c24761ce7fa956abe09d0925ad424b199ced7cca"}
Nov 03 19:47:53.227861 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.227828    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm" event={"ID":"a74714b9-b503-4817-b5d2-9ab74cd798a7","Type":"ContainerStarted","Data":"338c0c788bc3d388964e30c0c15dd488dddeadb72df4d0b439f20aefe9f111f4"}
Nov 03 19:47:53.227976 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.227884    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm" event={"ID":"a74714b9-b503-4817-b5d2-9ab74cd798a7","Type":"ContainerStarted","Data":"c5d45575c48ef1cbb626ba17f44ba0fff510d25d9cea039f3061421aad205d7d"}
Nov 03 19:47:53.227976 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.227935    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm"
Nov 03 19:47:53.230695 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.230671    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwn9t\" (UniqueName: \"kubernetes.io/projected/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-kube-api-access-jwn9t\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:53.230779 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.230701    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:53.230779 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.230719    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:53.230779 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.230733    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:53.235383 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.235366    5591 scope.go:122] "RemoveContainer" containerID="07bd397d52b25687427b76006cc0518d1c65812a4f209c3692e3491100e3b05f"
Nov 03 19:47:53.244357 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.244334    5591 scope.go:122] "RemoveContainer" containerID="9b593ef82c96e11ceb517d99ed48467d3d3b6dd7ffc592b7b292c2bdb518c6ec"
Nov 03 19:47:53.244646 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:47:53.244627    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b593ef82c96e11ceb517d99ed48467d3d3b6dd7ffc592b7b292c2bdb518c6ec\": container with ID starting with 9b593ef82c96e11ceb517d99ed48467d3d3b6dd7ffc592b7b292c2bdb518c6ec not found: ID does not exist" containerID="9b593ef82c96e11ceb517d99ed48467d3d3b6dd7ffc592b7b292c2bdb518c6ec"
Nov 03 19:47:53.244688 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.244656    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b593ef82c96e11ceb517d99ed48467d3d3b6dd7ffc592b7b292c2bdb518c6ec"} err="failed to get container status \"9b593ef82c96e11ceb517d99ed48467d3d3b6dd7ffc592b7b292c2bdb518c6ec\": rpc error: code = NotFound desc = could not find container \"9b593ef82c96e11ceb517d99ed48467d3d3b6dd7ffc592b7b292c2bdb518c6ec\": container with ID starting with 9b593ef82c96e11ceb517d99ed48467d3d3b6dd7ffc592b7b292c2bdb518c6ec not found: ID does not exist"
Nov 03 19:47:53.244688 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.244673    5591 scope.go:122] "RemoveContainer" containerID="07bd397d52b25687427b76006cc0518d1c65812a4f209c3692e3491100e3b05f"
Nov 03 19:47:53.244930 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:47:53.244912    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07bd397d52b25687427b76006cc0518d1c65812a4f209c3692e3491100e3b05f\": container with ID starting with 07bd397d52b25687427b76006cc0518d1c65812a4f209c3692e3491100e3b05f not found: ID does not exist" containerID="07bd397d52b25687427b76006cc0518d1c65812a4f209c3692e3491100e3b05f"
Nov 03 19:47:53.244967 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.244936    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07bd397d52b25687427b76006cc0518d1c65812a4f209c3692e3491100e3b05f"} err="failed to get container status \"07bd397d52b25687427b76006cc0518d1c65812a4f209c3692e3491100e3b05f\": rpc error: code = NotFound desc = could not find container \"07bd397d52b25687427b76006cc0518d1c65812a4f209c3692e3491100e3b05f\": container with ID starting with 07bd397d52b25687427b76006cc0518d1c65812a4f209c3692e3491100e3b05f not found: ID does not exist"
Nov 03 19:47:53.247042 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.246993    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm" podStartSLOduration=2.246977657 podStartE2EDuration="2.246977657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:53.246668998 +0000 UTC m=+622.018962018" watchObservedRunningTime="2026-11-03 19:47:53.246977657 +0000 UTC m=+622.019270678"
Nov 03 19:47:53.265134 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.265089    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb" podStartSLOduration=2.265077398 podStartE2EDuration="2.265077398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:53.264020806 +0000 UTC m=+622.036313822" watchObservedRunningTime="2026-11-03 19:47:53.265077398 +0000 UTC m=+622.037370429"
Nov 03 19:47:53.287850 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.287816    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"]
Nov 03 19:47:53.292186 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.292159    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c20ed791-openshift-default-7ff69c5dc9-l6z2f"]
Nov 03 19:47:53.348388 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.348350    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b7ef8f-17e3-4fe9-a98d-cbd45650f626" path="/var/lib/kubelet/pods/c7b7ef8f-17e3-4fe9-a98d-cbd45650f626/volumes"
Nov 03 19:47:53.355168 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.355142    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4"
Nov 03 19:47:53.638631 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.638581    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66"]
Nov 03 19:47:53.652144 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.648932    5591 cpu_manager.go:414] "containerMap: removing container" podUID="c7b7ef8f-17e3-4fe9-a98d-cbd45650f626" containerName="istio-proxy"
Nov 03 19:47:53.652144 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.648964    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b7ef8f-17e3-4fe9-a98d-cbd45650f626" containerName="istio-proxy"
Nov 03 19:47:53.652144 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.649007    5591 cpu_manager.go:414] "containerMap: removing container" podUID="c7b7ef8f-17e3-4fe9-a98d-cbd45650f626" containerName="istio-proxy"
Nov 03 19:47:53.652144 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.649033    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b7ef8f-17e3-4fe9-a98d-cbd45650f626" containerName="istio-proxy"
Nov 03 19:47:53.652144 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.649309    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b7ef8f-17e3-4fe9-a98d-cbd45650f626" containerName="istio-proxy"
Nov 03 19:47:53.667412 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.667380    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4"]
Nov 03 19:47:53.667531 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.667436    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66"]
Nov 03 19:47:53.667606 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.667548    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66"
Nov 03 19:47:53.693227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.690473    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-2b5703-serving-cert\""
Nov 03 19:47:53.851882 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.851847    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-metrics-certs\") pod \"maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66\" (UID: \"ddf2f97d-ab36-416b-9c5d-29330b70c4b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66"
Nov 03 19:47:53.852065 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.851984    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-maas-api-tls\") pod \"maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66\" (UID: \"ddf2f97d-ab36-416b-9c5d-29330b70c4b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66"
Nov 03 19:47:53.852065 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.852055    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxz4t\" (UniqueName: \"kubernetes.io/projected/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-kube-api-access-vxz4t\") pod \"maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66\" (UID: \"ddf2f97d-ab36-416b-9c5d-29330b70c4b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66"
Nov 03 19:47:53.852143 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.852108    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66\" (UID: \"ddf2f97d-ab36-416b-9c5d-29330b70c4b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66"
Nov 03 19:47:53.953699 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.953613    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxz4t\" (UniqueName: \"kubernetes.io/projected/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-kube-api-access-vxz4t\") pod \"maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66\" (UID: \"ddf2f97d-ab36-416b-9c5d-29330b70c4b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66"
Nov 03 19:47:53.953699 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.953674    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66\" (UID: \"ddf2f97d-ab36-416b-9c5d-29330b70c4b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66"
Nov 03 19:47:53.953887 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.953726    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-metrics-certs\") pod \"maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66\" (UID: \"ddf2f97d-ab36-416b-9c5d-29330b70c4b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66"
Nov 03 19:47:53.953887 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.953797    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-maas-api-tls\") pod \"maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66\" (UID: \"ddf2f97d-ab36-416b-9c5d-29330b70c4b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66"
Nov 03 19:47:53.954422 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.954392    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66\" (UID: \"ddf2f97d-ab36-416b-9c5d-29330b70c4b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66"
Nov 03 19:47:53.956398 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.956372    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-maas-api-tls\") pod \"maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66\" (UID: \"ddf2f97d-ab36-416b-9c5d-29330b70c4b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66"
Nov 03 19:47:53.956559 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.956539    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-metrics-certs\") pod \"maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66\" (UID: \"ddf2f97d-ab36-416b-9c5d-29330b70c4b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66"
Nov 03 19:47:53.970642 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:53.970620    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxz4t\" (UniqueName: \"kubernetes.io/projected/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-kube-api-access-vxz4t\") pod \"maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66\" (UID: \"ddf2f97d-ab36-416b-9c5d-29330b70c4b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66"
Nov 03 19:47:54.006785 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:54.006735    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66"
Nov 03 19:47:54.237462 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:54.237426    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4" event={"ID":"de96244c-7272-46a0-b20b-5cd1470fc8bc","Type":"ContainerStarted","Data":"788e1717bfc2ed8e2e903caec8f76a928a44183d343e3f142779c5a6713ed29b"}
Nov 03 19:47:54.237709 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:54.237679    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4" event={"ID":"de96244c-7272-46a0-b20b-5cd1470fc8bc","Type":"ContainerStarted","Data":"aa6783d61e97805283727f9bb532745f10b9db668408e5846f3b70a42db0185a"}
Nov 03 19:47:54.237812 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:54.237725    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4"
Nov 03 19:47:54.265180 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:54.265127    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4" podStartSLOduration=1.814737196 podStartE2EDuration="2.265111719s" totalImagesPullingTime="450.374523ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:54.262974789 +0000 UTC m=+623.035267834" watchObservedRunningTime="2026-11-03 19:47:54.265111719 +0000 UTC m=+623.037404737"
Nov 03 19:47:54.315320 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:54.315287    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66"]
Nov 03 19:47:54.315623 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:47:54.315598    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf2f97d_ab36_416b_9c5d_29330b70c4b4.slice/crio-7d80d7948f37ad06fce7a6356a73ba0f20cfed8a2e107651813fcfecb66979c2 WatchSource:0}: Error finding container 7d80d7948f37ad06fce7a6356a73ba0f20cfed8a2e107651813fcfecb66979c2: Status 404 returned error can't find the container with id 7d80d7948f37ad06fce7a6356a73ba0f20cfed8a2e107651813fcfecb66979c2
Nov 03 19:47:55.247264 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:55.247137    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66" event={"ID":"ddf2f97d-ab36-416b-9c5d-29330b70c4b4","Type":"ContainerStarted","Data":"646f5ac5a3ee21e60f965643a80ab6c763625996bf71b998503618de3746b524"}
Nov 03 19:47:55.247264 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:55.247225    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66" event={"ID":"ddf2f97d-ab36-416b-9c5d-29330b70c4b4","Type":"ContainerStarted","Data":"7d80d7948f37ad06fce7a6356a73ba0f20cfed8a2e107651813fcfecb66979c2"}
Nov 03 19:47:55.247764 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:55.247428    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66"
Nov 03 19:47:55.269381 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:55.269316    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66" podStartSLOduration=1.811859881 podStartE2EDuration="2.269294169s" totalImagesPullingTime="457.434288ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:55.267644157 +0000 UTC m=+624.039937172" watchObservedRunningTime="2026-11-03 19:47:55.269294169 +0000 UTC m=+624.041587191"
Nov 03 19:47:56.224193 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:56.224166    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j"
Nov 03 19:47:57.598549 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:57.598514    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv"]
Nov 03 19:47:57.613233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:57.609121    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b7ef8f-17e3-4fe9-a98d-cbd45650f626" containerName="istio-proxy"
Nov 03 19:47:57.622637 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:57.622605    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv"]
Nov 03 19:47:57.622935 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:57.622916    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv"
Nov 03 19:47:57.637713 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:57.637662    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-32e482-serving-cert\""
Nov 03 19:47:57.742947 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:57.742902    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhd4j\" (UniqueName: \"kubernetes.io/projected/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-kube-api-access-vhd4j\") pod \"maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv\" (UID: \"7e07d216-4ac1-414a-9803-a20d8a8bc2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv"
Nov 03 19:47:57.743130 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:57.742973    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv\" (UID: \"7e07d216-4ac1-414a-9803-a20d8a8bc2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv"
Nov 03 19:47:57.743130 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:57.742996    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-metrics-certs\") pod \"maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv\" (UID: \"7e07d216-4ac1-414a-9803-a20d8a8bc2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv"
Nov 03 19:47:57.743130 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:57.743064    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-maas-api-tls\") pod \"maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv\" (UID: \"7e07d216-4ac1-414a-9803-a20d8a8bc2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv"
Nov 03 19:47:57.843718 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:57.843682    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv\" (UID: \"7e07d216-4ac1-414a-9803-a20d8a8bc2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv"
Nov 03 19:47:57.843961 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:57.843945    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-metrics-certs\") pod \"maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv\" (UID: \"7e07d216-4ac1-414a-9803-a20d8a8bc2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv"
Nov 03 19:47:57.844176 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:57.844164    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-maas-api-tls\") pod \"maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv\" (UID: \"7e07d216-4ac1-414a-9803-a20d8a8bc2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv"
Nov 03 19:47:57.844325 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:57.844304    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhd4j\" (UniqueName: \"kubernetes.io/projected/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-kube-api-access-vhd4j\") pod \"maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv\" (UID: \"7e07d216-4ac1-414a-9803-a20d8a8bc2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv"
Nov 03 19:47:57.845003 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:57.844959    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv\" (UID: \"7e07d216-4ac1-414a-9803-a20d8a8bc2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv"
Nov 03 19:47:57.846914 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:57.846888    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-maas-api-tls\") pod \"maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv\" (UID: \"7e07d216-4ac1-414a-9803-a20d8a8bc2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv"
Nov 03 19:47:57.847157 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:57.847139    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-metrics-certs\") pod \"maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv\" (UID: \"7e07d216-4ac1-414a-9803-a20d8a8bc2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv"
Nov 03 19:47:57.859854 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:57.859780    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhd4j\" (UniqueName: \"kubernetes.io/projected/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-kube-api-access-vhd4j\") pod \"maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv\" (UID: \"7e07d216-4ac1-414a-9803-a20d8a8bc2c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv"
Nov 03 19:47:57.990742 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:57.990439    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv"
Nov 03 19:47:58.374430 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:58.374408    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv"]
Nov 03 19:47:58.762807 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:47:58.762723    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:47:59.271962 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:59.271930    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv" event={"ID":"7e07d216-4ac1-414a-9803-a20d8a8bc2c8","Type":"ContainerStarted","Data":"32e2539d9c51ecf98736ecadf4c59aeb8caa03ee8c196f1d861d74c62bcd8050"}
Nov 03 19:47:59.272148 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:59.271982    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv" event={"ID":"7e07d216-4ac1-414a-9803-a20d8a8bc2c8","Type":"ContainerStarted","Data":"7ed0723c6d73a728327ececfa6e53914504361ebb86b23a8f9b3e26d4999060b"}
Nov 03 19:47:59.272148 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:59.272062    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv"
Nov 03 19:47:59.298288 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:47:59.297264    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv" podStartSLOduration=1.653487698 podStartE2EDuration="2.297241387s" totalImagesPullingTime="643.753689ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:47:59.295637071 +0000 UTC m=+628.067930090" watchObservedRunningTime="2026-11-03 19:47:59.297241387 +0000 UTC m=+628.069534406"
Nov 03 19:48:00.028676 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:00.028642    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:48:00.253688 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:00.253656    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4"
Nov 03 19:48:01.258949 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:01.258921    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66"
Nov 03 19:48:02.220276 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:02.220247    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh"
Nov 03 19:48:03.212712 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:03.209469    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:48:03.212712 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:03.209584    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r" podUID="8e00f1a1-8156-4fa7-8308-b48127f82aa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:48:04.242570 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:04.242536    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb"
Nov 03 19:48:04.243007 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:04.242737    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm"
Nov 03 19:48:04.324616 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:04.324581    5591 generic.go:356] "Generic (PLEG): container finished" podID="8e00f1a1-8156-4fa7-8308-b48127f82aa6" containerID="da60d0a13ec638f966a0e24803738a66382b122c2968c8495a32494fe926366e" exitCode=0
Nov 03 19:48:04.324794 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:04.324640    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r" event={"ID":"8e00f1a1-8156-4fa7-8308-b48127f82aa6","Type":"ContainerDied","Data":"da60d0a13ec638f966a0e24803738a66382b122c2968c8495a32494fe926366e"}
Nov 03 19:48:04.325063 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:04.325046    5591 scope.go:122] "RemoveContainer" containerID="da60d0a13ec638f966a0e24803738a66382b122c2968c8495a32494fe926366e"
Nov 03 19:48:04.326659 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:04.326624    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:48:04.326779 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:04.326723    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:48:04.326843 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:04.326774    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:48:05.308657 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:05.308631    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv"
Nov 03 19:48:05.333397 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:05.333360    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r" event={"ID":"8e00f1a1-8156-4fa7-8308-b48127f82aa6","Type":"ContainerStarted","Data":"6b6268803ecbc18a4e4ac07fc79f3ad4034cc8ff312a3cdefe50da49f8e83cd0"}
Nov 03 19:48:05.629569 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:05.629545    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:48:05.634851 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:05.629639    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 19:48:05.634851 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:05.629692    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r" podUID="8e00f1a1-8156-4fa7-8308-b48127f82aa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 19:48:06.339953 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:06.339916    5591 generic.go:356] "Generic (PLEG): container finished" podID="1faa21e4-5bf7-418d-adc8-3fc5bf955010" containerID="382ad4b105b08bb53ffbcbad01efa24da8a59243875b635ae171dba58a67a149" exitCode=0
Nov 03 19:48:06.340445 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:06.339991    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989" event={"ID":"1faa21e4-5bf7-418d-adc8-3fc5bf955010","Type":"ContainerDied","Data":"382ad4b105b08bb53ffbcbad01efa24da8a59243875b635ae171dba58a67a149"}
Nov 03 19:48:06.340550 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:06.340535    5591 scope.go:122] "RemoveContainer" containerID="382ad4b105b08bb53ffbcbad01efa24da8a59243875b635ae171dba58a67a149"
Nov 03 19:48:06.341705 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:06.341675    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:48:06.341761 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:06.341749    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:48:06.341796 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:06.341780    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:48:06.629159 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:06.629068    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 19:48:06.629159 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:06.629143    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r" podUID="8e00f1a1-8156-4fa7-8308-b48127f82aa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 19:48:07.348943 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:07.348906    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989" event={"ID":"1faa21e4-5bf7-418d-adc8-3fc5bf955010","Type":"ContainerStarted","Data":"90f25ed1b925a5da58add530a5b9b74af4d57ab007e8898b05ce218141167469"}
Nov 03 19:48:07.632899 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:07.632807    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 19:48:07.633169 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:07.633143    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r" podUID="8e00f1a1-8156-4fa7-8308-b48127f82aa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 19:48:08.290609 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:08.290381    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:48:08.291357 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:08.291330    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 03 19:48:08.291888 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:08.291549    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989" podUID="1faa21e4-5bf7-418d-adc8-3fc5bf955010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 03 19:48:08.631692 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:08.631584    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 19:48:08.631692 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:08.631655    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r" podUID="8e00f1a1-8156-4fa7-8308-b48127f82aa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 19:48:09.232253 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:09.228697    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:48:09.293245 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:09.290630    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 03 19:48:09.293245 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:09.290693    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989" podUID="1faa21e4-5bf7-418d-adc8-3fc5bf955010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 03 19:48:09.634428 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:09.634375    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 19:48:09.635037 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:09.634997    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r" podUID="8e00f1a1-8156-4fa7-8308-b48127f82aa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 19:48:10.294777 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:10.294735    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 03 19:48:10.295596 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:10.295563    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989" podUID="1faa21e4-5bf7-418d-adc8-3fc5bf955010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 03 19:48:10.752862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:10.752785    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:48:10.753514 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:10.753406    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:48:10.754650 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:10.754620    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 19:48:11.291351 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:11.291313    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 03 19:48:11.291643 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:11.291611    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989" podUID="1faa21e4-5bf7-418d-adc8-3fc5bf955010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 03 19:48:12.300229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:12.299676    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Nov 03 19:48:12.300229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:12.299754    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989" podUID="1faa21e4-5bf7-418d-adc8-3fc5bf955010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Nov 03 19:48:13.294475 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:13.294444    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:48:13.294869 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:13.294844    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:48:13.295547 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:13.295528    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 19:48:18.438241 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:18.438194    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:48:26.922001 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:26.921968    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:48:27.846755 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:27.846726    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"]
Nov 03 19:48:27.851198 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:27.851173    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:48:27.861137 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:27.861111    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"kube-root-ca.crt\""
Nov 03 19:48:27.861346 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:27.861231    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"default-dockercfg-zrxnc\""
Nov 03 19:48:27.861346 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:27.861235    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"premium-w3-9f46e5-kserve-self-signed-certs\""
Nov 03 19:48:27.861599 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:27.861575    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"openshift-service-ca.crt\""
Nov 03 19:48:27.863726 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:27.863705    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"]
Nov 03 19:48:27.908804 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:27.908775    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-dshm\") pod \"premium-w3-9f46e5-kserve-66b569d97d-g8hq4\" (UID: \"8cf0c23f-8862-442a-a487-35f4023789b8\") " pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:48:27.908989 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:27.908835    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-model-cache\") pod \"premium-w3-9f46e5-kserve-66b569d97d-g8hq4\" (UID: \"8cf0c23f-8862-442a-a487-35f4023789b8\") " pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:48:27.908989 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:27.908936    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-tmp-dir\") pod \"premium-w3-9f46e5-kserve-66b569d97d-g8hq4\" (UID: \"8cf0c23f-8862-442a-a487-35f4023789b8\") " pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:48:27.908989 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:27.908984    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cf0c23f-8862-442a-a487-35f4023789b8-tls-certs\") pod \"premium-w3-9f46e5-kserve-66b569d97d-g8hq4\" (UID: \"8cf0c23f-8862-442a-a487-35f4023789b8\") " pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:48:27.909096 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:27.909003    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz79h\" (UniqueName: \"kubernetes.io/projected/8cf0c23f-8862-442a-a487-35f4023789b8-kube-api-access-vz79h\") pod \"premium-w3-9f46e5-kserve-66b569d97d-g8hq4\" (UID: \"8cf0c23f-8862-442a-a487-35f4023789b8\") " pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:48:27.909096 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:27.909074    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-home\") pod \"premium-w3-9f46e5-kserve-66b569d97d-g8hq4\" (UID: \"8cf0c23f-8862-442a-a487-35f4023789b8\") " pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:48:28.012150 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:28.012102    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-dshm\") pod \"premium-w3-9f46e5-kserve-66b569d97d-g8hq4\" (UID: \"8cf0c23f-8862-442a-a487-35f4023789b8\") " pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:48:28.012695 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:28.012262    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-model-cache\") pod \"premium-w3-9f46e5-kserve-66b569d97d-g8hq4\" (UID: \"8cf0c23f-8862-442a-a487-35f4023789b8\") " pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:48:28.012695 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:28.012324    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-tmp-dir\") pod \"premium-w3-9f46e5-kserve-66b569d97d-g8hq4\" (UID: \"8cf0c23f-8862-442a-a487-35f4023789b8\") " pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:48:28.012695 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:28.012365    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cf0c23f-8862-442a-a487-35f4023789b8-tls-certs\") pod \"premium-w3-9f46e5-kserve-66b569d97d-g8hq4\" (UID: \"8cf0c23f-8862-442a-a487-35f4023789b8\") " pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:48:28.012695 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:28.012387    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz79h\" (UniqueName: \"kubernetes.io/projected/8cf0c23f-8862-442a-a487-35f4023789b8-kube-api-access-vz79h\") pod \"premium-w3-9f46e5-kserve-66b569d97d-g8hq4\" (UID: \"8cf0c23f-8862-442a-a487-35f4023789b8\") " pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:48:28.012695 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:28.012408    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-home\") pod \"premium-w3-9f46e5-kserve-66b569d97d-g8hq4\" (UID: \"8cf0c23f-8862-442a-a487-35f4023789b8\") " pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:48:28.012972 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:28.012854    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-home\") pod \"premium-w3-9f46e5-kserve-66b569d97d-g8hq4\" (UID: \"8cf0c23f-8862-442a-a487-35f4023789b8\") " pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:48:28.012972 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:28.012950    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-model-cache\") pod \"premium-w3-9f46e5-kserve-66b569d97d-g8hq4\" (UID: \"8cf0c23f-8862-442a-a487-35f4023789b8\") " pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:48:28.013084 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:28.013072    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-tmp-dir\") pod \"premium-w3-9f46e5-kserve-66b569d97d-g8hq4\" (UID: \"8cf0c23f-8862-442a-a487-35f4023789b8\") " pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:48:28.015501 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:28.015471    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-dshm\") pod \"premium-w3-9f46e5-kserve-66b569d97d-g8hq4\" (UID: \"8cf0c23f-8862-442a-a487-35f4023789b8\") " pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:48:28.015737 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:28.015713    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cf0c23f-8862-442a-a487-35f4023789b8-tls-certs\") pod \"premium-w3-9f46e5-kserve-66b569d97d-g8hq4\" (UID: \"8cf0c23f-8862-442a-a487-35f4023789b8\") " pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:48:28.030139 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:28.030111    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz79h\" (UniqueName: \"kubernetes.io/projected/8cf0c23f-8862-442a-a487-35f4023789b8-kube-api-access-vz79h\") pod \"premium-w3-9f46e5-kserve-66b569d97d-g8hq4\" (UID: \"8cf0c23f-8862-442a-a487-35f4023789b8\") " pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:48:28.163307 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:28.163191    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:48:28.441351 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:28.441326    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"]
Nov 03 19:48:28.443660 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:48:28.443633    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8cf0c23f_8862_442a_a487_35f4023789b8.slice/crio-c0955f6febd1005f9b4623fcd96bdc84bc3d8a8e971d550567df829f7c1ca4c7 WatchSource:0}: Error finding container c0955f6febd1005f9b4623fcd96bdc84bc3d8a8e971d550567df829f7c1ca4c7: Status 404 returned error can't find the container with id c0955f6febd1005f9b4623fcd96bdc84bc3d8a8e971d550567df829f7c1ca4c7
Nov 03 19:48:28.491136 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:28.491003    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" event={"ID":"8cf0c23f-8862-442a-a487-35f4023789b8","Type":"ContainerStarted","Data":"c0955f6febd1005f9b4623fcd96bdc84bc3d8a8e971d550567df829f7c1ca4c7"}
Nov 03 19:48:28.724843 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:48:28.724701    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:48:29.498343 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:29.498304    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" event={"ID":"8cf0c23f-8862-442a-a487-35f4023789b8","Type":"ContainerStarted","Data":"e3b01a953a5f0a40565252edadff3c23cb9b38a7481930778869d0e03761c66d"}
Nov 03 19:48:29.517441 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:29.517395    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podStartSLOduration=2.517380293 podStartE2EDuration="2.517380293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:29.516285848 +0000 UTC m=+658.288578865" watchObservedRunningTime="2026-11-03 19:48:29.517380293 +0000 UTC m=+658.289673311"
Nov 03 19:48:30.163934 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:30.163905    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:48:38.137967 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:38.137931    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:48:38.163555 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:38.163522    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:48:38.176993 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:38.176960    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:48:38.605444 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:38.605413    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:48:43.283949 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:43.283909    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"]
Nov 03 19:48:43.289822 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:43.289802    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:48:43.298744 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:43.298717    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2b5703\"/\"kube-root-ca.crt\""
Nov 03 19:48:43.298890 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:43.298795    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2b5703\"/\"default-dockercfg-4rsdq\""
Nov 03 19:48:43.298890 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:43.298827    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2b5703\"/\"openshift-service-ca.crt\""
Nov 03 19:48:43.298890 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:43.298843    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2b5703\"/\"premium-w2-2b5703-kserve-self-signed-certs\""
Nov 03 19:48:43.301164 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:43.301143    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"]
Nov 03 19:48:43.373258 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:43.373220    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-dshm\") pod \"premium-w2-2b5703-kserve-69744b97b7-m68tz\" (UID: \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\") " pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:48:43.373424 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:43.373319    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h78gf\" (UniqueName: \"kubernetes.io/projected/58f1605e-5db2-4a8d-9b62-4c84e113ac33-kube-api-access-h78gf\") pod \"premium-w2-2b5703-kserve-69744b97b7-m68tz\" (UID: \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\") " pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:48:43.373424 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:43.373353    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58f1605e-5db2-4a8d-9b62-4c84e113ac33-tls-certs\") pod \"premium-w2-2b5703-kserve-69744b97b7-m68tz\" (UID: \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\") " pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:48:43.373424 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:43.373374    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-tmp-dir\") pod \"premium-w2-2b5703-kserve-69744b97b7-m68tz\" (UID: \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\") " pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:48:43.373558 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:43.373523    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-home\") pod \"premium-w2-2b5703-kserve-69744b97b7-m68tz\" (UID: \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\") " pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:48:43.373601 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:43.373569    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-model-cache\") pod \"premium-w2-2b5703-kserve-69744b97b7-m68tz\" (UID: \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\") " pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:48:43.474698 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:43.474661    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h78gf\" (UniqueName: \"kubernetes.io/projected/58f1605e-5db2-4a8d-9b62-4c84e113ac33-kube-api-access-h78gf\") pod \"premium-w2-2b5703-kserve-69744b97b7-m68tz\" (UID: \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\") " pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:48:43.474938 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:43.474707    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58f1605e-5db2-4a8d-9b62-4c84e113ac33-tls-certs\") pod \"premium-w2-2b5703-kserve-69744b97b7-m68tz\" (UID: \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\") " pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:48:43.474938 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:43.474732    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-tmp-dir\") pod \"premium-w2-2b5703-kserve-69744b97b7-m68tz\" (UID: \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\") " pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:48:43.474938 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:43.474778    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-home\") pod \"premium-w2-2b5703-kserve-69744b97b7-m68tz\" (UID: \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\") " pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:48:43.474938 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:43.474845    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-model-cache\") pod \"premium-w2-2b5703-kserve-69744b97b7-m68tz\" (UID: \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\") " pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:48:43.474938 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:43.474889    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-dshm\") pod \"premium-w2-2b5703-kserve-69744b97b7-m68tz\" (UID: \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\") " pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:48:43.475326 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:43.475300    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-tmp-dir\") pod \"premium-w2-2b5703-kserve-69744b97b7-m68tz\" (UID: \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\") " pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:48:43.475326 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:43.475313    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-home\") pod \"premium-w2-2b5703-kserve-69744b97b7-m68tz\" (UID: \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\") " pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:48:43.475453 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:43.475361    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-model-cache\") pod \"premium-w2-2b5703-kserve-69744b97b7-m68tz\" (UID: \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\") " pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:48:43.477229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:43.477189    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-dshm\") pod \"premium-w2-2b5703-kserve-69744b97b7-m68tz\" (UID: \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\") " pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:48:43.477399 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:43.477381    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58f1605e-5db2-4a8d-9b62-4c84e113ac33-tls-certs\") pod \"premium-w2-2b5703-kserve-69744b97b7-m68tz\" (UID: \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\") " pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:48:43.487030 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:43.487010    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h78gf\" (UniqueName: \"kubernetes.io/projected/58f1605e-5db2-4a8d-9b62-4c84e113ac33-kube-api-access-h78gf\") pod \"premium-w2-2b5703-kserve-69744b97b7-m68tz\" (UID: \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\") " pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:48:43.604445 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:43.604357    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:48:43.888001 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:43.887974    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"]
Nov 03 19:48:43.890459 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:48:43.890427    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58f1605e_5db2_4a8d_9b62_4c84e113ac33.slice/crio-6d15c4e829b760a48de4f49a570fcd27e527c556856bb45fcb0d6086e937700d WatchSource:0}: Error finding container 6d15c4e829b760a48de4f49a570fcd27e527c556856bb45fcb0d6086e937700d: Status 404 returned error can't find the container with id 6d15c4e829b760a48de4f49a570fcd27e527c556856bb45fcb0d6086e937700d
Nov 03 19:48:44.589192 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:44.589142    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" event={"ID":"58f1605e-5db2-4a8d-9b62-4c84e113ac33","Type":"ContainerStarted","Data":"debd15accce6666c5ffb2b5dacffb9f4a02cc4706e520987ed1a0fa8c7002839"}
Nov 03 19:48:44.589192 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:44.589217    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" event={"ID":"58f1605e-5db2-4a8d-9b62-4c84e113ac33","Type":"ContainerStarted","Data":"6d15c4e829b760a48de4f49a570fcd27e527c556856bb45fcb0d6086e937700d"}
Nov 03 19:48:44.604932 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:44.604889    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:48:44.607872 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:44.607823    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podStartSLOduration=1.6078059439999999 podStartE2EDuration="1.607805944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:44.607047646 +0000 UTC m=+673.379340665" watchObservedRunningTime="2026-11-03 19:48:44.607805944 +0000 UTC m=+673.380098962"
Nov 03 19:48:48.886903 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:48.886867    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"]
Nov 03 19:48:48.891078 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:48.891053    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:48.897603 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:48.897578    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-32e482-openshift-default-dockercfg-5pv2v\""
Nov 03 19:48:48.903605 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:48.903582    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"]
Nov 03 19:48:48.937087 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:48.937052    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:48.937087 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:48.937086    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d7456ba-061d-4c8a-974c-326c72027557-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:48.937402 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:48.937111    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d7456ba-061d-4c8a-974c-326c72027557-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:48.937402 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:48.937283    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk6rp\" (UniqueName: \"kubernetes.io/projected/6d7456ba-061d-4c8a-974c-326c72027557-kube-api-access-qk6rp\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:48.937481 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:48.937404    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:48.937481 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:48.937438    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d7456ba-061d-4c8a-974c-326c72027557-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:48.937481 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:48.937475    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:48.937640 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:48.937566    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:48.937705 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:48.937687    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:49.039192 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:49.039158    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:49.039192 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:49.039193    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d7456ba-061d-4c8a-974c-326c72027557-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:49.039471 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:49.039240    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:49.039471 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:49.039264    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:49.039471 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:49.039321    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:49.039471 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:49.039370    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:49.039471 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:49.039386    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d7456ba-061d-4c8a-974c-326c72027557-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:49.039471 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:49.039407    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d7456ba-061d-4c8a-974c-326c72027557-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:49.039471 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:49.039428    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk6rp\" (UniqueName: \"kubernetes.io/projected/6d7456ba-061d-4c8a-974c-326c72027557-kube-api-access-qk6rp\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:49.039811 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:49.039662    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:49.039811 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:49.039694    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:49.039811 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:49.039783    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:49.040045 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:49.040022    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:49.040145 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:49.040126    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d7456ba-061d-4c8a-974c-326c72027557-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:49.041856 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:49.041833    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:49.042232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:49.042190    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d7456ba-061d-4c8a-974c-326c72027557-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:49.051781 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:49.051753    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d7456ba-061d-4c8a-974c-326c72027557-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:49.051952 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:49.051932    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk6rp\" (UniqueName: \"kubernetes.io/projected/6d7456ba-061d-4c8a-974c-326c72027557-kube-api-access-qk6rp\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:49.207434 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:49.207333    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:49.494089 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:49.494063    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"]
Nov 03 19:48:49.496375 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:48:49.496349    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d7456ba_061d_4c8a_974c_326c72027557.slice/crio-dc34fdacf9b8533e3f02bc8e6ffed2ff9e66000c1d75f814e8f0a66be76623d1 WatchSource:0}: Error finding container dc34fdacf9b8533e3f02bc8e6ffed2ff9e66000c1d75f814e8f0a66be76623d1: Status 404 returned error can't find the container with id dc34fdacf9b8533e3f02bc8e6ffed2ff9e66000c1d75f814e8f0a66be76623d1
Nov 03 19:48:49.498736 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:49.498704    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:48:49.498814 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:49.498779    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:48:49.498814 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:49.498810    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:48:49.619098 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:49.619064    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v" event={"ID":"6d7456ba-061d-4c8a-974c-326c72027557","Type":"ContainerStarted","Data":"0f19d6863d124322e0a6ffba90bd2d002acc852418bdd319f483f6a8b900a5f4"}
Nov 03 19:48:49.619314 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:49.619121    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v" event={"ID":"6d7456ba-061d-4c8a-974c-326c72027557","Type":"ContainerStarted","Data":"dc34fdacf9b8533e3f02bc8e6ffed2ff9e66000c1d75f814e8f0a66be76623d1"}
Nov 03 19:48:49.642396 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:49.642333    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v" podStartSLOduration=1.6423138050000001 podStartE2EDuration="1.642313805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:49.64109662 +0000 UTC m=+678.413389664" watchObservedRunningTime="2026-11-03 19:48:49.642313805 +0000 UTC m=+678.414606825"
Nov 03 19:48:50.141321 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:50.141282    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:48:50.208388 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:50.208353    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:50.209963 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:50.209935    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 19:48:50.210090 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:50.209989    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v" podUID="6d7456ba-061d-4c8a-974c-326c72027557" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 19:48:51.211750 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:51.211715    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 19:48:51.212403 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:51.212356    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v" podUID="6d7456ba-061d-4c8a-974c-326c72027557" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 19:48:52.208042 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:52.207999    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 19:48:52.208288 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:52.208060    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v" podUID="6d7456ba-061d-4c8a-974c-326c72027557" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 19:48:53.207740 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:53.207706    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 19:48:53.208141 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:53.207760    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v" podUID="6d7456ba-061d-4c8a-974c-326c72027557" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 19:48:53.604872 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:53.604824    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:48:53.620331 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:53.620304    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:48:53.656616 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:53.656583    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:48:54.207955 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:54.207916    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 19:48:54.208406 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:54.207982    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v" podUID="6d7456ba-061d-4c8a-974c-326c72027557" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 19:48:55.208674 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:55.208641    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 19:48:55.209065 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:55.208713    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v" podUID="6d7456ba-061d-4c8a-974c-326c72027557" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 19:48:56.212672 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:56.212643    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:56.213098 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:56.213006    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:56.213729 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:48:56.213707    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 19:48:58.724889 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:48:58.724853    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:49:10.344001 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:10.343963    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:49:16.310152 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.310119    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"]
Nov 03 19:49:16.314130 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.314110    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.318355 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.318329    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d3d0df-openshift-default-dockercfg-m4lvp\""
Nov 03 19:49:16.327061 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.327035    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"]
Nov 03 19:49:16.384239 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.384190    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.384445 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.384250    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vbp8\" (UniqueName: \"kubernetes.io/projected/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-kube-api-access-8vbp8\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.384445 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.384321    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.384445 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.384393    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.384615 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.384458    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.384615 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.384485    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.384743 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.384621    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.384743 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.384658    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.384833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.384780    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.485786 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.485742    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.486669 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.486639    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vbp8\" (UniqueName: \"kubernetes.io/projected/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-kube-api-access-8vbp8\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.487642 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.487619    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.488079 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.488059    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.489080 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.489058    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.490255 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.490233    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.490789 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.490771    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.491248 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.491228    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.491985 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.491936    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.492946 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.491154    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.493142 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.489946    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.493265 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.488171    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.493385 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.492754    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.493492 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.488617    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.498778 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.498753    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.499635 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.499611    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.500117 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.500096    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vbp8\" (UniqueName: \"kubernetes.io/projected/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-kube-api-access-8vbp8\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.502594 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.502571    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.628607 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.628522    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:16.917847 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.917822    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"]
Nov 03 19:49:16.920337 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:49:16.920310    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f1ccac_bfc0_4b66_a445_86b3c40afdfe.slice/crio-6e7cd03665329ba0df26ff8db19a6db1a169e9e92654ef504a4f5837e556bcab WatchSource:0}: Error finding container 6e7cd03665329ba0df26ff8db19a6db1a169e9e92654ef504a4f5837e556bcab: Status 404 returned error can't find the container with id 6e7cd03665329ba0df26ff8db19a6db1a169e9e92654ef504a4f5837e556bcab
Nov 03 19:49:16.923037 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.922991    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:49:16.923133 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.923094    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:49:16.923175 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:16.923137    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:49:17.806175 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:17.806137    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv" event={"ID":"53f1ccac-bfc0-4b66-a445-86b3c40afdfe","Type":"ContainerStarted","Data":"c7f47954a4d434316a07bc4297b472ebb2324292c4dd68766b961faecbfd691a"}
Nov 03 19:49:17.806714 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:17.806275    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv" event={"ID":"53f1ccac-bfc0-4b66-a445-86b3c40afdfe","Type":"ContainerStarted","Data":"6e7cd03665329ba0df26ff8db19a6db1a169e9e92654ef504a4f5837e556bcab"}
Nov 03 19:49:17.831935 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:17.831884    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv" podStartSLOduration=1.831869816 podStartE2EDuration="1.831869816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:17.829806613 +0000 UTC m=+706.602099632" watchObservedRunningTime="2026-11-03 19:49:17.831869816 +0000 UTC m=+706.604162837"
Nov 03 19:49:18.629640 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:18.629609    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:18.631418 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:18.631391    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 19:49:18.631492 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:18.631447    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv" podUID="53f1ccac-bfc0-4b66-a445-86b3c40afdfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 19:49:19.631694 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:19.631654    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 19:49:19.632273 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:19.631718    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv" podUID="53f1ccac-bfc0-4b66-a445-86b3c40afdfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 19:49:20.629688 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:20.629659    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 19:49:20.629875 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:20.629713    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv" podUID="53f1ccac-bfc0-4b66-a445-86b3c40afdfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 19:49:21.629435 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:21.629388    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 19:49:21.629842 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:21.629455    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv" podUID="53f1ccac-bfc0-4b66-a445-86b3c40afdfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 19:49:22.629324 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:22.629282    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 19:49:22.629521 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:22.629337    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv" podUID="53f1ccac-bfc0-4b66-a445-86b3c40afdfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 19:49:23.633473 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:23.633436    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:23.633865 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:23.633713    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:23.634538 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:23.634518    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 19:49:28.723905 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:49:28.723870    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:49:49.039680 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.039558    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-9cvzn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:49:49.039680 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.039640    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn" podUID="3a0f28bc-d05f-45bb-8431-3aaf73529510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:49:49.216441 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.216328    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"]
Nov 03 19:49:49.220577 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.220551    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.226251 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.226183    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3fbb1a-openshift-default-dockercfg-247ct\""
Nov 03 19:49:49.238645 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.238617    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"]
Nov 03 19:49:49.327717 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.327625    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.327717 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.327658    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9thn\" (UniqueName: \"kubernetes.io/projected/e31559fd-2ec2-48cd-8f00-8167a567097b-kube-api-access-h9thn\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.327945 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.327766    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.327945 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.327832    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.327945 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.327882    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.327945 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.327937    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e31559fd-2ec2-48cd-8f00-8167a567097b-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.328124 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.327968    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.328124 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.327997    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.328124 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.328099    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.429167 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.429130    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e31559fd-2ec2-48cd-8f00-8167a567097b-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.429371 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.429177    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.429371 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.429223    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.429371 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.429265    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.429371 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.429301    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.429371 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.429325    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9thn\" (UniqueName: \"kubernetes.io/projected/e31559fd-2ec2-48cd-8f00-8167a567097b-kube-api-access-h9thn\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.429610 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.429381    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.429610 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.429429    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.429610 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.429462    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.429775 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.429719    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.429859 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.429835    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.429989 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.429972    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.430047 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.430004    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.430047 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.430015    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e31559fd-2ec2-48cd-8f00-8167a567097b-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.431932 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.431909    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.432228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.432191    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.445180 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.445139    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.445330 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.445245    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9thn\" (UniqueName: \"kubernetes.io/projected/e31559fd-2ec2-48cd-8f00-8167a567097b-kube-api-access-h9thn\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.543214 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.543174    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:49.831655 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.831629    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"]
Nov 03 19:49:49.833632 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:49:49.833608    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode31559fd_2ec2_48cd_8f00_8167a567097b.slice/crio-d5f314e297403c0645026e2ff759cef301903b54bfeddebc014e1761c4b9305e WatchSource:0}: Error finding container d5f314e297403c0645026e2ff759cef301903b54bfeddebc014e1761c4b9305e: Status 404 returned error can't find the container with id d5f314e297403c0645026e2ff759cef301903b54bfeddebc014e1761c4b9305e
Nov 03 19:49:49.835900 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.835866    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:49:49.835997 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.835933    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:49:49.835997 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:49.835980    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:49:50.063852 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:50.063812    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm" event={"ID":"e31559fd-2ec2-48cd-8f00-8167a567097b","Type":"ContainerStarted","Data":"0d4e4f3915c407f348d4baf6f79380faa66d002ca74f21e14d930c0248bd2501"}
Nov 03 19:49:50.064348 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:50.063874    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm" event={"ID":"e31559fd-2ec2-48cd-8f00-8167a567097b","Type":"ContainerStarted","Data":"d5f314e297403c0645026e2ff759cef301903b54bfeddebc014e1761c4b9305e"}
Nov 03 19:49:50.087255 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:50.087175    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm" podStartSLOduration=1.087158477 podStartE2EDuration="1.087158477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:50.085947023 +0000 UTC m=+738.858240043" watchObservedRunningTime="2026-11-03 19:49:50.087158477 +0000 UTC m=+738.859451494"
Nov 03 19:49:50.543437 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:50.543408    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:50.545226 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:50.545181    5591 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:49:50.545353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:50.545243    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm" podUID="e31559fd-2ec2-48cd-8f00-8167a567097b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 19:49:51.543777 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:51.543738    5591 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:49:51.544172 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:51.543827    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm" podUID="e31559fd-2ec2-48cd-8f00-8167a567097b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 19:49:52.543721 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:52.543634    5591 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:49:52.543721 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:52.543697    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm" podUID="e31559fd-2ec2-48cd-8f00-8167a567097b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 19:49:53.543773 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:53.543741    5591 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:49:53.543946 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:53.543805    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm" podUID="e31559fd-2ec2-48cd-8f00-8167a567097b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 19:49:54.543712 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:54.543676    5591 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:49:54.543894 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:54.543743    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm" podUID="e31559fd-2ec2-48cd-8f00-8167a567097b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 19:49:55.544467 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:55.544431    5591 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 19:49:55.544946 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:55.544495    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm" podUID="e31559fd-2ec2-48cd-8f00-8167a567097b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 19:49:56.211657 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:56.211621    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:49:56.549169 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:56.549127    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:56.549674 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:56.549544    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:56.550345 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:49:56.550326    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 19:49:58.742580 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:49:58.742485    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:50:01.032294 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:01.030024    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4"]
Nov 03 19:50:01.057668 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:01.057634    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4"
Nov 03 19:50:01.080455 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:01.080425    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-bs8pz\""
Nov 03 19:50:01.080817 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:01.080475    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 19:50:01.080954 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:01.080548    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b46da6fce236\""
Nov 03 19:50:01.081047 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:01.080584    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 19:50:01.081140 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:01.080663    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 19:50:01.081271 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:01.080761    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 19:50:01.126907 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:01.126857    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4"]
Nov 03 19:50:01.229274 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:01.229241    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/564d28e9-7533-471d-babf-70e6032d2fb0-plugins-config-volume\") pod \"payload-processing-external-model-559cf8ccdf-ljnr4\" (UID: \"564d28e9-7533-471d-babf-70e6032d2fb0\") " pod="models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4"
Nov 03 19:50:01.229464 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:01.229338    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/564d28e9-7533-471d-babf-70e6032d2fb0-provider-credentials\") pod \"payload-processing-external-model-559cf8ccdf-ljnr4\" (UID: \"564d28e9-7533-471d-babf-70e6032d2fb0\") " pod="models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4"
Nov 03 19:50:01.229529 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:01.229506    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/564d28e9-7533-471d-babf-70e6032d2fb0-routing-overlay-volume\") pod \"payload-processing-external-model-559cf8ccdf-ljnr4\" (UID: \"564d28e9-7533-471d-babf-70e6032d2fb0\") " pod="models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4"
Nov 03 19:50:01.229646 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:01.229592    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5g92\" (UniqueName: \"kubernetes.io/projected/564d28e9-7533-471d-babf-70e6032d2fb0-kube-api-access-j5g92\") pod \"payload-processing-external-model-559cf8ccdf-ljnr4\" (UID: \"564d28e9-7533-471d-babf-70e6032d2fb0\") " pod="models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4"
Nov 03 19:50:01.331171 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:01.331092    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/564d28e9-7533-471d-babf-70e6032d2fb0-routing-overlay-volume\") pod \"payload-processing-external-model-559cf8ccdf-ljnr4\" (UID: \"564d28e9-7533-471d-babf-70e6032d2fb0\") " pod="models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4"
Nov 03 19:50:01.331171 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:01.331143    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5g92\" (UniqueName: \"kubernetes.io/projected/564d28e9-7533-471d-babf-70e6032d2fb0-kube-api-access-j5g92\") pod \"payload-processing-external-model-559cf8ccdf-ljnr4\" (UID: \"564d28e9-7533-471d-babf-70e6032d2fb0\") " pod="models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4"
Nov 03 19:50:01.331425 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:01.331188    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/564d28e9-7533-471d-babf-70e6032d2fb0-plugins-config-volume\") pod \"payload-processing-external-model-559cf8ccdf-ljnr4\" (UID: \"564d28e9-7533-471d-babf-70e6032d2fb0\") " pod="models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4"
Nov 03 19:50:01.331425 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:01.331246    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/564d28e9-7533-471d-babf-70e6032d2fb0-provider-credentials\") pod \"payload-processing-external-model-559cf8ccdf-ljnr4\" (UID: \"564d28e9-7533-471d-babf-70e6032d2fb0\") " pod="models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4"
Nov 03 19:50:01.331953 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:01.331924    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/564d28e9-7533-471d-babf-70e6032d2fb0-routing-overlay-volume\") pod \"payload-processing-external-model-559cf8ccdf-ljnr4\" (UID: \"564d28e9-7533-471d-babf-70e6032d2fb0\") " pod="models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4"
Nov 03 19:50:01.332121 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:01.332100    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/564d28e9-7533-471d-babf-70e6032d2fb0-plugins-config-volume\") pod \"payload-processing-external-model-559cf8ccdf-ljnr4\" (UID: \"564d28e9-7533-471d-babf-70e6032d2fb0\") " pod="models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4"
Nov 03 19:50:01.333786 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:01.333763    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/564d28e9-7533-471d-babf-70e6032d2fb0-provider-credentials\") pod \"payload-processing-external-model-559cf8ccdf-ljnr4\" (UID: \"564d28e9-7533-471d-babf-70e6032d2fb0\") " pod="models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4"
Nov 03 19:50:01.344175 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:01.344110    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5g92\" (UniqueName: \"kubernetes.io/projected/564d28e9-7533-471d-babf-70e6032d2fb0-kube-api-access-j5g92\") pod \"payload-processing-external-model-559cf8ccdf-ljnr4\" (UID: \"564d28e9-7533-471d-babf-70e6032d2fb0\") " pod="models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4"
Nov 03 19:50:01.429013 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:01.428980    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4"
Nov 03 19:50:01.715386 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:01.715360    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4"]
Nov 03 19:50:01.717502 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:50:01.717475    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod564d28e9_7533_471d_babf_70e6032d2fb0.slice/crio-05abd359dd5d668e011ecf5aa4ced50779e59890b13699572aa954a1ddcd9931 WatchSource:0}: Error finding container 05abd359dd5d668e011ecf5aa4ced50779e59890b13699572aa954a1ddcd9931: Status 404 returned error can't find the container with id 05abd359dd5d668e011ecf5aa4ced50779e59890b13699572aa954a1ddcd9931
Nov 03 19:50:02.162465 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:02.162423    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4" event={"ID":"564d28e9-7533-471d-babf-70e6032d2fb0","Type":"ContainerStarted","Data":"d0481593a7b5b14cb4dc368f45bc5ff80678e5c0a52843326c16e59dc6f020db"}
Nov 03 19:50:02.163115 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:02.162493    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4" event={"ID":"564d28e9-7533-471d-babf-70e6032d2fb0","Type":"ContainerStarted","Data":"05abd359dd5d668e011ecf5aa4ced50779e59890b13699572aa954a1ddcd9931"}
Nov 03 19:50:02.163115 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:02.162572    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4"
Nov 03 19:50:02.187729 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:02.187680    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4" podStartSLOduration=1.187666454 podStartE2EDuration="1.187666454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:02.184964012 +0000 UTC m=+750.957257031" watchObservedRunningTime="2026-11-03 19:50:02.187666454 +0000 UTC m=+750.959959467"
Nov 03 19:50:09.307864 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.307824    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"]
Nov 03 19:50:09.313600 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.313575    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.358543 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.358512    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"]
Nov 03 19:50:09.370954 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.370915    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c03ef4-fb42-4a08-b02d-98165eb9b203-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-zst5d\" (UID: \"e8c03ef4-fb42-4a08-b02d-98165eb9b203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.371129 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.370963    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8c03ef4-fb42-4a08-b02d-98165eb9b203-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-zst5d\" (UID: \"e8c03ef4-fb42-4a08-b02d-98165eb9b203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.371129 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.371038    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb2zk\" (UniqueName: \"kubernetes.io/projected/e8c03ef4-fb42-4a08-b02d-98165eb9b203-kube-api-access-qb2zk\") pod \"maas-default-gateway-openshift-default-855f9574cf-zst5d\" (UID: \"e8c03ef4-fb42-4a08-b02d-98165eb9b203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.371235 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.371137    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c03ef4-fb42-4a08-b02d-98165eb9b203-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-zst5d\" (UID: \"e8c03ef4-fb42-4a08-b02d-98165eb9b203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.371235 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.371196    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c03ef4-fb42-4a08-b02d-98165eb9b203-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-zst5d\" (UID: \"e8c03ef4-fb42-4a08-b02d-98165eb9b203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.371318 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.371267    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c03ef4-fb42-4a08-b02d-98165eb9b203-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-zst5d\" (UID: \"e8c03ef4-fb42-4a08-b02d-98165eb9b203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.371318 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.371301    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c03ef4-fb42-4a08-b02d-98165eb9b203-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-zst5d\" (UID: \"e8c03ef4-fb42-4a08-b02d-98165eb9b203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.371412 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.371366    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c03ef4-fb42-4a08-b02d-98165eb9b203-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-zst5d\" (UID: \"e8c03ef4-fb42-4a08-b02d-98165eb9b203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.371576 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.371549    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c03ef4-fb42-4a08-b02d-98165eb9b203-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-zst5d\" (UID: \"e8c03ef4-fb42-4a08-b02d-98165eb9b203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.386425 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.384603    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"]
Nov 03 19:50:09.390680 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.390651    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.435000 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.434963    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"]
Nov 03 19:50:09.472330 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.472299    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c03ef4-fb42-4a08-b02d-98165eb9b203-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-zst5d\" (UID: \"e8c03ef4-fb42-4a08-b02d-98165eb9b203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.472330 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.472334    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c03ef4-fb42-4a08-b02d-98165eb9b203-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-zst5d\" (UID: \"e8c03ef4-fb42-4a08-b02d-98165eb9b203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.472627 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.472357    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c03ef4-fb42-4a08-b02d-98165eb9b203-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-zst5d\" (UID: \"e8c03ef4-fb42-4a08-b02d-98165eb9b203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.472627 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.472376    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-t8m7q\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.472627 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.472405    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-t8m7q\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.472627 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.472560    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-t8m7q\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.472627 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.472613    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7r8g\" (UniqueName: \"kubernetes.io/projected/68b82f0c-e037-4366-891b-1ee898b2cba2-kube-api-access-s7r8g\") pod \"maas-default-gateway-openshift-default-855f9574cf-t8m7q\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.472914 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.472674    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b82f0c-e037-4366-891b-1ee898b2cba2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-t8m7q\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.472914 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.472720    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c03ef4-fb42-4a08-b02d-98165eb9b203-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-zst5d\" (UID: \"e8c03ef4-fb42-4a08-b02d-98165eb9b203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.472914 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.472794    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-t8m7q\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.472914 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.472846    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-t8m7q\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.472914 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.472862    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c03ef4-fb42-4a08-b02d-98165eb9b203-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-zst5d\" (UID: \"e8c03ef4-fb42-4a08-b02d-98165eb9b203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.472914 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.472893    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c03ef4-fb42-4a08-b02d-98165eb9b203-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-zst5d\" (UID: \"e8c03ef4-fb42-4a08-b02d-98165eb9b203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.472914 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.472847    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c03ef4-fb42-4a08-b02d-98165eb9b203-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-zst5d\" (UID: \"e8c03ef4-fb42-4a08-b02d-98165eb9b203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.473318 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.473163    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c03ef4-fb42-4a08-b02d-98165eb9b203-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-zst5d\" (UID: \"e8c03ef4-fb42-4a08-b02d-98165eb9b203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.473318 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.473220    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8c03ef4-fb42-4a08-b02d-98165eb9b203-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-zst5d\" (UID: \"e8c03ef4-fb42-4a08-b02d-98165eb9b203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.473318 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.473167    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c03ef4-fb42-4a08-b02d-98165eb9b203-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-zst5d\" (UID: \"e8c03ef4-fb42-4a08-b02d-98165eb9b203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.473318 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.473264    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-t8m7q\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.473580 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.473367    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-t8m7q\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.473580 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.473407    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb2zk\" (UniqueName: \"kubernetes.io/projected/e8c03ef4-fb42-4a08-b02d-98165eb9b203-kube-api-access-qb2zk\") pod \"maas-default-gateway-openshift-default-855f9574cf-zst5d\" (UID: \"e8c03ef4-fb42-4a08-b02d-98165eb9b203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.473580 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.473436    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c03ef4-fb42-4a08-b02d-98165eb9b203-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-zst5d\" (UID: \"e8c03ef4-fb42-4a08-b02d-98165eb9b203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.473580 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.473555    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c03ef4-fb42-4a08-b02d-98165eb9b203-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-zst5d\" (UID: \"e8c03ef4-fb42-4a08-b02d-98165eb9b203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.473867 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.473842    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c03ef4-fb42-4a08-b02d-98165eb9b203-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-zst5d\" (UID: \"e8c03ef4-fb42-4a08-b02d-98165eb9b203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.484236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.476312    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c03ef4-fb42-4a08-b02d-98165eb9b203-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-zst5d\" (UID: \"e8c03ef4-fb42-4a08-b02d-98165eb9b203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.484236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.476585    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c03ef4-fb42-4a08-b02d-98165eb9b203-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-zst5d\" (UID: \"e8c03ef4-fb42-4a08-b02d-98165eb9b203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.498593 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.498513    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb2zk\" (UniqueName: \"kubernetes.io/projected/e8c03ef4-fb42-4a08-b02d-98165eb9b203-kube-api-access-qb2zk\") pod \"maas-default-gateway-openshift-default-855f9574cf-zst5d\" (UID: \"e8c03ef4-fb42-4a08-b02d-98165eb9b203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.502454 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.502382    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8c03ef4-fb42-4a08-b02d-98165eb9b203-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-zst5d\" (UID: \"e8c03ef4-fb42-4a08-b02d-98165eb9b203\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.574487 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.574411    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-t8m7q\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.574487 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.574449    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-t8m7q\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.574685 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.574491    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-t8m7q\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.574685 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.574550    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7r8g\" (UniqueName: \"kubernetes.io/projected/68b82f0c-e037-4366-891b-1ee898b2cba2-kube-api-access-s7r8g\") pod \"maas-default-gateway-openshift-default-855f9574cf-t8m7q\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.574685 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.574584    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b82f0c-e037-4366-891b-1ee898b2cba2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-t8m7q\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.574685 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.574648    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-t8m7q\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.574817 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.574690    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-t8m7q\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.574817 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.574764    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-t8m7q\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.574817 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.574796    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-t8m7q\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.575093 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.575068    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-t8m7q\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.575252 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.575116    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-t8m7q\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.575252 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.575152    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-t8m7q\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.575581 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.575558    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-t8m7q\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.575867 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.575847    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b82f0c-e037-4366-891b-1ee898b2cba2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-t8m7q\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.619227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.601650    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-t8m7q\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.619227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.606684    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-t8m7q\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.619227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.614429    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-t8m7q\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.619227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.618891    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7r8g\" (UniqueName: \"kubernetes.io/projected/68b82f0c-e037-4366-891b-1ee898b2cba2-kube-api-access-s7r8g\") pod \"maas-default-gateway-openshift-default-855f9574cf-t8m7q\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.658494 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.658465    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:09.712554 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.712304    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:09.974582 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.974554    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"]
Nov 03 19:50:10.027272 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.989039    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:50:10.027272 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.989129    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:50:10.027272 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:09.989182    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:50:10.060859 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:10.060831    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"]
Nov 03 19:50:10.062693 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:50:10.062665    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b82f0c_e037_4366_891b_1ee898b2cba2.slice/crio-e112b9e913494860fcce82bbe22b77e12b947106b9e4f9676e934205c2576d04 WatchSource:0}: Error finding container e112b9e913494860fcce82bbe22b77e12b947106b9e4f9676e934205c2576d04: Status 404 returned error can't find the container with id e112b9e913494860fcce82bbe22b77e12b947106b9e4f9676e934205c2576d04
Nov 03 19:50:10.065232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:10.065176    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:50:10.065311 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:10.065267    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:50:10.065311 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:10.065301    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:50:10.381280 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:10.357261    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q" event={"ID":"68b82f0c-e037-4366-891b-1ee898b2cba2","Type":"ContainerStarted","Data":"64533ad18d753c4cf532a3950904c061dff824d13699ca5de5480a4e40535b11"}
Nov 03 19:50:10.381280 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:10.357359    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q" event={"ID":"68b82f0c-e037-4366-891b-1ee898b2cba2","Type":"ContainerStarted","Data":"e112b9e913494860fcce82bbe22b77e12b947106b9e4f9676e934205c2576d04"}
Nov 03 19:50:10.413286 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:10.401795    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d" event={"ID":"e8c03ef4-fb42-4a08-b02d-98165eb9b203","Type":"ContainerStarted","Data":"a2396a6ef3bb1fecec3daa9f5c13c69bc88f2fceaafe0448dbeeab8cc5c10a9f"}
Nov 03 19:50:10.413286 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:10.401859    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d" event={"ID":"e8c03ef4-fb42-4a08-b02d-98165eb9b203","Type":"ContainerStarted","Data":"80d4bfaf0b1509741ffd6baa84ee6641ca9ba054f3ad0de93f67c5e460dc246f"}
Nov 03 19:50:10.439229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:10.415420    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q" podStartSLOduration=1.4153987049999999 podStartE2EDuration="1.415398705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:10.403017703 +0000 UTC m=+759.175310723" watchObservedRunningTime="2026-11-03 19:50:10.415398705 +0000 UTC m=+759.187691794"
Nov 03 19:50:10.446238 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:10.440141    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d" podStartSLOduration=1.440121945 podStartE2EDuration="1.440121945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:10.439680207 +0000 UTC m=+759.211973250" watchObservedRunningTime="2026-11-03 19:50:10.440121945 +0000 UTC m=+759.212414966"
Nov 03 19:50:10.659327 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:10.659290    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:10.715489 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:10.715385    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:11.662549 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:11.662366    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-zst5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:50:11.662549 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:11.662430    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d" podUID="e8c03ef4-fb42-4a08-b02d-98165eb9b203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 19:50:11.715227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:11.715168    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-t8m7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 19:50:11.715473 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:11.715447    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q" podUID="68b82f0c-e037-4366-891b-1ee898b2cba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 19:50:12.659544 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:12.659496    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-zst5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:50:12.659756 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:12.659564    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d" podUID="e8c03ef4-fb42-4a08-b02d-98165eb9b203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 19:50:12.719367 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:12.719327    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-t8m7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 19:50:12.719967 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:12.719936    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q" podUID="68b82f0c-e037-4366-891b-1ee898b2cba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 19:50:13.219232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:13.218267    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4"
Nov 03 19:50:13.659096 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:13.659061    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-zst5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:50:13.659329 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:13.659120    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d" podUID="e8c03ef4-fb42-4a08-b02d-98165eb9b203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 19:50:13.720233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:13.716622    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-t8m7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 19:50:13.720233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:13.716684    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q" podUID="68b82f0c-e037-4366-891b-1ee898b2cba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 19:50:14.617074 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:14.617033    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:50:14.659116 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:14.659083    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-zst5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:50:14.659300 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:14.659153    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d" podUID="e8c03ef4-fb42-4a08-b02d-98165eb9b203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 19:50:14.719396 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:14.719361    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-t8m7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 19:50:14.719689 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:14.719663    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q" podUID="68b82f0c-e037-4366-891b-1ee898b2cba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 19:50:15.659614 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:15.659578    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-zst5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:50:15.660198 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:15.660166    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d" podUID="e8c03ef4-fb42-4a08-b02d-98165eb9b203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 19:50:15.716630 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:15.716593    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-t8m7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 19:50:15.716920 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:15.716891    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q" podUID="68b82f0c-e037-4366-891b-1ee898b2cba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 19:50:16.659153 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:16.659124    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-zst5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:50:16.659331 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:16.659187    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d" podUID="e8c03ef4-fb42-4a08-b02d-98165eb9b203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 19:50:16.713316 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:16.713268    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-t8m7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 19:50:16.713713 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:16.713365    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q" podUID="68b82f0c-e037-4366-891b-1ee898b2cba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 19:50:17.659330 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:17.659298    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-zst5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:50:17.659491 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:17.659360    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d" podUID="e8c03ef4-fb42-4a08-b02d-98165eb9b203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 19:50:17.713321 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:17.713290    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-t8m7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 03 19:50:17.713799 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:17.713348    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q" podUID="68b82f0c-e037-4366-891b-1ee898b2cba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 03 19:50:19.213496 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:19.213462    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:19.214580 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:19.214552    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:19.215735 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:19.215712    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d"
Nov 03 19:50:19.384162 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:19.383616    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:19.384461 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:19.384439    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:19.385351 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:19.385332    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 19:50:20.364139 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:20.364105    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t"]
Nov 03 19:50:20.368558 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:20.368534    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t"
Nov 03 19:50:20.375463 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:20.375438    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ee59ddb9f8a1\""
Nov 03 19:50:20.379162 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:20.379138    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t"]
Nov 03 19:50:20.555597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:20.555564    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2168bf74-6936-46ea-b16a-d73af51825ee-plugins-config-volume\") pod \"payload-processing-external-model-65d589b9db-w6g9t\" (UID: \"2168bf74-6936-46ea-b16a-d73af51825ee\") " pod="models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t"
Nov 03 19:50:20.555797 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:20.555618    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4v9c\" (UniqueName: \"kubernetes.io/projected/2168bf74-6936-46ea-b16a-d73af51825ee-kube-api-access-m4v9c\") pod \"payload-processing-external-model-65d589b9db-w6g9t\" (UID: \"2168bf74-6936-46ea-b16a-d73af51825ee\") " pod="models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t"
Nov 03 19:50:20.555860 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:20.555791    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2168bf74-6936-46ea-b16a-d73af51825ee-provider-credentials\") pod \"payload-processing-external-model-65d589b9db-w6g9t\" (UID: \"2168bf74-6936-46ea-b16a-d73af51825ee\") " pod="models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t"
Nov 03 19:50:20.555860 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:20.555851    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2168bf74-6936-46ea-b16a-d73af51825ee-routing-overlay-volume\") pod \"payload-processing-external-model-65d589b9db-w6g9t\" (UID: \"2168bf74-6936-46ea-b16a-d73af51825ee\") " pod="models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t"
Nov 03 19:50:20.656654 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:20.656565    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2168bf74-6936-46ea-b16a-d73af51825ee-provider-credentials\") pod \"payload-processing-external-model-65d589b9db-w6g9t\" (UID: \"2168bf74-6936-46ea-b16a-d73af51825ee\") " pod="models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t"
Nov 03 19:50:20.656841 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:20.656658    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2168bf74-6936-46ea-b16a-d73af51825ee-routing-overlay-volume\") pod \"payload-processing-external-model-65d589b9db-w6g9t\" (UID: \"2168bf74-6936-46ea-b16a-d73af51825ee\") " pod="models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t"
Nov 03 19:50:20.656841 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:20.656735    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2168bf74-6936-46ea-b16a-d73af51825ee-plugins-config-volume\") pod \"payload-processing-external-model-65d589b9db-w6g9t\" (UID: \"2168bf74-6936-46ea-b16a-d73af51825ee\") " pod="models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t"
Nov 03 19:50:20.656966 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:20.656867    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4v9c\" (UniqueName: \"kubernetes.io/projected/2168bf74-6936-46ea-b16a-d73af51825ee-kube-api-access-m4v9c\") pod \"payload-processing-external-model-65d589b9db-w6g9t\" (UID: \"2168bf74-6936-46ea-b16a-d73af51825ee\") " pod="models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t"
Nov 03 19:50:20.657463 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:20.657439    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2168bf74-6936-46ea-b16a-d73af51825ee-routing-overlay-volume\") pod \"payload-processing-external-model-65d589b9db-w6g9t\" (UID: \"2168bf74-6936-46ea-b16a-d73af51825ee\") " pod="models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t"
Nov 03 19:50:20.657568 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:20.657542    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2168bf74-6936-46ea-b16a-d73af51825ee-plugins-config-volume\") pod \"payload-processing-external-model-65d589b9db-w6g9t\" (UID: \"2168bf74-6936-46ea-b16a-d73af51825ee\") " pod="models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t"
Nov 03 19:50:20.659428 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:20.659408    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2168bf74-6936-46ea-b16a-d73af51825ee-provider-credentials\") pod \"payload-processing-external-model-65d589b9db-w6g9t\" (UID: \"2168bf74-6936-46ea-b16a-d73af51825ee\") " pod="models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t"
Nov 03 19:50:20.670072 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:20.670052    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4v9c\" (UniqueName: \"kubernetes.io/projected/2168bf74-6936-46ea-b16a-d73af51825ee-kube-api-access-m4v9c\") pod \"payload-processing-external-model-65d589b9db-w6g9t\" (UID: \"2168bf74-6936-46ea-b16a-d73af51825ee\") " pod="models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t"
Nov 03 19:50:20.684989 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:20.684961    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t"
Nov 03 19:50:20.969020 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:20.968995    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t"]
Nov 03 19:50:20.971268 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:50:20.971241    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2168bf74_6936_46ea_b16a_d73af51825ee.slice/crio-2358f7e85b1b5af5f740be9efbb0ed2d7e60069c459aa1d561310d0b0a2c1f69 WatchSource:0}: Error finding container 2358f7e85b1b5af5f740be9efbb0ed2d7e60069c459aa1d561310d0b0a2c1f69: Status 404 returned error can't find the container with id 2358f7e85b1b5af5f740be9efbb0ed2d7e60069c459aa1d561310d0b0a2c1f69
Nov 03 19:50:21.496623 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:21.496582    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t" event={"ID":"2168bf74-6936-46ea-b16a-d73af51825ee","Type":"ContainerStarted","Data":"adeeedc9ba24f92a81219b95c794e7dac11d5337fd97aad809a9463b90a8804c"}
Nov 03 19:50:21.497050 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:21.496648    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t"
Nov 03 19:50:21.497050 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:21.496670    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t" event={"ID":"2168bf74-6936-46ea-b16a-d73af51825ee","Type":"ContainerStarted","Data":"2358f7e85b1b5af5f740be9efbb0ed2d7e60069c459aa1d561310d0b0a2c1f69"}
Nov 03 19:50:21.561450 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:21.561386    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t" podStartSLOduration=1.5613682789999999 podStartE2EDuration="1.561368279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:21.561199067 +0000 UTC m=+770.333492086" watchObservedRunningTime="2026-11-03 19:50:21.561368279 +0000 UTC m=+770.333661299"
Nov 03 19:50:28.723869 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:50:28.723827    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:50:28.957397 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:28.957365    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:50:32.517428 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:32.517394    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t"
Nov 03 19:50:32.600273 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:32.599809    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4"]
Nov 03 19:50:34.058934 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.058901    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"]
Nov 03 19:50:34.064306 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.064287    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.079070 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.079047    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"]
Nov 03 19:50:34.108531 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.108504    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.108671 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.108536    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.108671 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.108560    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.108793 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.108735    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.108870 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.108848    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.108926 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.108894    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a223486f-1753-4dd1-8fd5-a6c03d664555-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.108982 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.108964    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.109038 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.109021    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfrcq\" (UniqueName: \"kubernetes.io/projected/a223486f-1753-4dd1-8fd5-a6c03d664555-kube-api-access-wfrcq\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.109084 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.109065    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.209724 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.209689    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfrcq\" (UniqueName: \"kubernetes.io/projected/a223486f-1753-4dd1-8fd5-a6c03d664555-kube-api-access-wfrcq\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.209724 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.209726    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.209957 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.209788    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.209957 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.209804    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.209957 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.209824    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.210121 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.210035    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.210243 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.210191    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.210344 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.210294    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.210344 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.210306    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.210548 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.210528    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.210695 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.210667    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a223486f-1753-4dd1-8fd5-a6c03d664555-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.210834 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.210740    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.211386 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.211357    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.214752 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.211372    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a223486f-1753-4dd1-8fd5-a6c03d664555-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.219563 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.219541    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.220004 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.219981    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.225025 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.225006    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfrcq\" (UniqueName: \"kubernetes.io/projected/a223486f-1753-4dd1-8fd5-a6c03d664555-kube-api-access-wfrcq\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.227387 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.227364    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.308655 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.308619    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"]
Nov 03 19:50:34.314136 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.314075    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.333390 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.333364    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"]
Nov 03 19:50:34.377810 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.377781    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:34.412693 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.412652    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c70d023-1700-42d0-964f-5573383d1355-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.412861 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.412702    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.413059 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.413037    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c70d023-1700-42d0-964f-5573383d1355-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.413136 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.413073    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grzpd\" (UniqueName: \"kubernetes.io/projected/3c70d023-1700-42d0-964f-5573383d1355-kube-api-access-grzpd\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.413136 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.413108    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.413305 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.413147    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.413305 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.413170    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.413305 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.413263    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c70d023-1700-42d0-964f-5573383d1355-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.413305 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.413281    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.514284 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.514246    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c70d023-1700-42d0-964f-5573383d1355-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.514436 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.514292    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.514436 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.514351    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c70d023-1700-42d0-964f-5573383d1355-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.514436 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.514378    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grzpd\" (UniqueName: \"kubernetes.io/projected/3c70d023-1700-42d0-964f-5573383d1355-kube-api-access-grzpd\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.514436 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.514428    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.514630 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.514453    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.514630 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.514475    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.514630 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.514539    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c70d023-1700-42d0-964f-5573383d1355-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.514630 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.514560    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.514859 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.514837    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.514951 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.514925    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.514990 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.514934    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.515044 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.515016    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.515178 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.515061    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c70d023-1700-42d0-964f-5573383d1355-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.516778 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.516750    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.517175 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.517155    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c70d023-1700-42d0-964f-5573383d1355-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.527024 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.526998    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c70d023-1700-42d0-964f-5573383d1355-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.527324 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.527304    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grzpd\" (UniqueName: \"kubernetes.io/projected/3c70d023-1700-42d0-964f-5573383d1355-kube-api-access-grzpd\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.628863 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.628780    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:34.674631 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.674601    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"]
Nov 03 19:50:34.681013 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.680974    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:50:34.681147 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.681066    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:50:34.681147 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.681140    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:50:34.925665 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.925637    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"]
Nov 03 19:50:34.928219 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:50:34.928170    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c70d023_1700_42d0_964f_5573383d1355.slice/crio-092f3cc7c453afc123104c3797c5d8d6a06ab4db4debde0b506a78e6233b3e19 WatchSource:0}: Error finding container 092f3cc7c453afc123104c3797c5d8d6a06ab4db4debde0b506a78e6233b3e19: Status 404 returned error can't find the container with id 092f3cc7c453afc123104c3797c5d8d6a06ab4db4debde0b506a78e6233b3e19
Nov 03 19:50:34.931286 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.931249    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:50:34.931398 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.931340    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:50:34.931398 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:34.931391    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:50:35.674965 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:35.674931    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s" event={"ID":"a223486f-1753-4dd1-8fd5-a6c03d664555","Type":"ContainerStarted","Data":"223d0287fb8ef34d6dd45d94490ec148c5e07cfd2b40f839e25e7de690956c5e"}
Nov 03 19:50:35.685689 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:35.685653    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s" event={"ID":"a223486f-1753-4dd1-8fd5-a6c03d664555","Type":"ContainerStarted","Data":"650b92723bc563b0e89f2af94921237342fa083e50a95a450fbb40bc25ad0e4c"}
Nov 03 19:50:35.695193 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:35.695158    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7" event={"ID":"3c70d023-1700-42d0-964f-5573383d1355","Type":"ContainerStarted","Data":"456c1f059ed2fdcef06f23f5cf106c7ca251c32337d4d7794f0d6c13e96268b9"}
Nov 03 19:50:35.695507 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:35.695466    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7" event={"ID":"3c70d023-1700-42d0-964f-5573383d1355","Type":"ContainerStarted","Data":"092f3cc7c453afc123104c3797c5d8d6a06ab4db4debde0b506a78e6233b3e19"}
Nov 03 19:50:35.743228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:35.742897    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s" podStartSLOduration=1.742875611 podStartE2EDuration="1.742875611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:35.717398522 +0000 UTC m=+784.489691541" watchObservedRunningTime="2026-11-03 19:50:35.742875611 +0000 UTC m=+784.515168631"
Nov 03 19:50:35.772779 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:35.772724    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7" podStartSLOduration=1.772710035 podStartE2EDuration="1.772710035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:35.772038009 +0000 UTC m=+784.544331026" watchObservedRunningTime="2026-11-03 19:50:35.772710035 +0000 UTC m=+784.545003044"
Nov 03 19:50:36.399229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:36.396842    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:50:36.399229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:36.396906    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s" podUID="a223486f-1753-4dd1-8fd5-a6c03d664555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 19:50:36.399229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:36.397180    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:36.636233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:36.636135    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:36.637641 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:36.637608    5591 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:50:36.637768 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:36.637663    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7" podUID="3c70d023-1700-42d0-964f-5573383d1355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 19:50:37.388233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:37.386595    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:50:37.388233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:37.386655    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s" podUID="a223486f-1753-4dd1-8fd5-a6c03d664555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 19:50:37.613230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:37.603531    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4" podUID="564d28e9-7533-471d-babf-70e6032d2fb0" containerName="payload-processing" containerID="cri-o://d0481593a7b5b14cb4dc368f45bc5ff80678e5c0a52843326c16e59dc6f020db" gracePeriod=25
Nov 03 19:50:37.634070 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:37.631700    5591 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:50:37.634070 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:37.631776    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7" podUID="3c70d023-1700-42d0-964f-5573383d1355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 19:50:37.964377 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:37.964354    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4"
Nov 03 19:50:37.973854 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:37.973832    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/564d28e9-7533-471d-babf-70e6032d2fb0-kube-api-access-j5g92\" (UniqueName: \"kubernetes.io/projected/564d28e9-7533-471d-babf-70e6032d2fb0-kube-api-access-j5g92\") pod \"564d28e9-7533-471d-babf-70e6032d2fb0\" (UID: \"564d28e9-7533-471d-babf-70e6032d2fb0\") "
Nov 03 19:50:37.974035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:37.973979    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/564d28e9-7533-471d-babf-70e6032d2fb0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/564d28e9-7533-471d-babf-70e6032d2fb0-provider-credentials\") pod \"564d28e9-7533-471d-babf-70e6032d2fb0\" (UID: \"564d28e9-7533-471d-babf-70e6032d2fb0\") "
Nov 03 19:50:37.974035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:37.973998    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/564d28e9-7533-471d-babf-70e6032d2fb0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/564d28e9-7533-471d-babf-70e6032d2fb0-routing-overlay-volume\") pod \"564d28e9-7533-471d-babf-70e6032d2fb0\" (UID: \"564d28e9-7533-471d-babf-70e6032d2fb0\") "
Nov 03 19:50:37.974110 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:37.974036    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/564d28e9-7533-471d-babf-70e6032d2fb0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/564d28e9-7533-471d-babf-70e6032d2fb0-plugins-config-volume\") pod \"564d28e9-7533-471d-babf-70e6032d2fb0\" (UID: \"564d28e9-7533-471d-babf-70e6032d2fb0\") "
Nov 03 19:50:37.974517 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:37.974495    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/564d28e9-7533-471d-babf-70e6032d2fb0-routing-overlay-volume" pod "564d28e9-7533-471d-babf-70e6032d2fb0" (UID: "564d28e9-7533-471d-babf-70e6032d2fb0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:50:37.974613 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:37.974554    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/564d28e9-7533-471d-babf-70e6032d2fb0-plugins-config-volume" pod "564d28e9-7533-471d-babf-70e6032d2fb0" (UID: "564d28e9-7533-471d-babf-70e6032d2fb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:50:37.981414 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:37.981388    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/564d28e9-7533-471d-babf-70e6032d2fb0-provider-credentials" pod "564d28e9-7533-471d-babf-70e6032d2fb0" (UID: "564d28e9-7533-471d-babf-70e6032d2fb0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:50:38.004732 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:38.000345    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/564d28e9-7533-471d-babf-70e6032d2fb0-kube-api-access-j5g92" pod "564d28e9-7533-471d-babf-70e6032d2fb0" (UID: "564d28e9-7533-471d-babf-70e6032d2fb0"). InnerVolumeSpecName "kube-api-access-j5g92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:50:38.075612 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:38.075581    5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/564d28e9-7533-471d-babf-70e6032d2fb0-provider-credentials\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:38.075612 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:38.075610    5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/564d28e9-7533-471d-babf-70e6032d2fb0-routing-overlay-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:38.075612 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:38.075621    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/564d28e9-7533-471d-babf-70e6032d2fb0-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:38.075612 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:38.075630    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5g92\" (UniqueName: \"kubernetes.io/projected/564d28e9-7533-471d-babf-70e6032d2fb0-kube-api-access-j5g92\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:38.380727 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:38.380687    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:50:38.381045 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:38.381015    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s" podUID="a223486f-1753-4dd1-8fd5-a6c03d664555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 19:50:38.635298 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:38.631557    5591 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:50:38.635298 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:38.631615    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7" podUID="3c70d023-1700-42d0-964f-5573383d1355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 19:50:38.741840 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:38.741813    5591 generic.go:356] "Generic (PLEG): container finished" podID="564d28e9-7533-471d-babf-70e6032d2fb0" containerID="d0481593a7b5b14cb4dc368f45bc5ff80678e5c0a52843326c16e59dc6f020db" exitCode=0
Nov 03 19:50:38.742506 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:38.741869    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4"
Nov 03 19:50:38.742506 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:38.741897    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4" event={"ID":"564d28e9-7533-471d-babf-70e6032d2fb0","Type":"ContainerDied","Data":"d0481593a7b5b14cb4dc368f45bc5ff80678e5c0a52843326c16e59dc6f020db"}
Nov 03 19:50:38.742506 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:38.742159    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4" event={"ID":"564d28e9-7533-471d-babf-70e6032d2fb0","Type":"ContainerDied","Data":"05abd359dd5d668e011ecf5aa4ced50779e59890b13699572aa954a1ddcd9931"}
Nov 03 19:50:38.742506 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:38.742180    5591 scope.go:122] "RemoveContainer" containerID="d0481593a7b5b14cb4dc368f45bc5ff80678e5c0a52843326c16e59dc6f020db"
Nov 03 19:50:38.760623 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:38.760604    5591 scope.go:122] "RemoveContainer" containerID="d0481593a7b5b14cb4dc368f45bc5ff80678e5c0a52843326c16e59dc6f020db"
Nov 03 19:50:38.760918 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:50:38.760896    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0481593a7b5b14cb4dc368f45bc5ff80678e5c0a52843326c16e59dc6f020db\": container with ID starting with d0481593a7b5b14cb4dc368f45bc5ff80678e5c0a52843326c16e59dc6f020db not found: ID does not exist" containerID="d0481593a7b5b14cb4dc368f45bc5ff80678e5c0a52843326c16e59dc6f020db"
Nov 03 19:50:38.760966 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:38.760927    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0481593a7b5b14cb4dc368f45bc5ff80678e5c0a52843326c16e59dc6f020db"} err="failed to get container status \"d0481593a7b5b14cb4dc368f45bc5ff80678e5c0a52843326c16e59dc6f020db\": rpc error: code = NotFound desc = could not find container \"d0481593a7b5b14cb4dc368f45bc5ff80678e5c0a52843326c16e59dc6f020db\": container with ID starting with d0481593a7b5b14cb4dc368f45bc5ff80678e5c0a52843326c16e59dc6f020db not found: ID does not exist"
Nov 03 19:50:38.769537 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:38.769511    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4"]
Nov 03 19:50:38.776042 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:38.776014    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-559cf8ccdf-ljnr4"]
Nov 03 19:50:39.348854 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:39.348822    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="564d28e9-7533-471d-babf-70e6032d2fb0" path="/var/lib/kubelet/pods/564d28e9-7533-471d-babf-70e6032d2fb0/volumes"
Nov 03 19:50:39.386623 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:39.386590    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:50:39.386897 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:39.386868    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s" podUID="a223486f-1753-4dd1-8fd5-a6c03d664555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 19:50:39.642290 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:39.636153    5591 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:50:39.642290 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:39.636268    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7" podUID="3c70d023-1700-42d0-964f-5573383d1355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 19:50:40.139257 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.137368    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"]
Nov 03 19:50:40.139257 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.138183    5591 cpu_manager.go:414] "containerMap: removing container" podUID="564d28e9-7533-471d-babf-70e6032d2fb0" containerName="payload-processing"
Nov 03 19:50:40.139257 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.138229    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="564d28e9-7533-471d-babf-70e6032d2fb0" containerName="payload-processing"
Nov 03 19:50:40.139257 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.138412    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="564d28e9-7533-471d-babf-70e6032d2fb0" containerName="payload-processing"
Nov 03 19:50:40.146030 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.145704    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"]
Nov 03 19:50:40.146030 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.145810    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:50:40.151057 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.150723    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"openshift-service-ca.crt\""
Nov 03 19:50:40.151806 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.151767    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"kube-root-ca.crt\""
Nov 03 19:50:40.151926 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.151911    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"default-dockercfg-lxnx2\""
Nov 03 19:50:40.152029 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.151990    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"distinct-w4-3fbb1a-kserve-self-signed-certs\""
Nov 03 19:50:40.245258 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.244010    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-tls-certs\") pod \"distinct-w4-3fbb1a-kserve-69d65db78b-thc7w\" (UID: \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:50:40.245258 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.244060    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-model-cache\") pod \"distinct-w4-3fbb1a-kserve-69d65db78b-thc7w\" (UID: \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:50:40.245258 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.244085    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfpht\" (UniqueName: \"kubernetes.io/projected/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-kube-api-access-cfpht\") pod \"distinct-w4-3fbb1a-kserve-69d65db78b-thc7w\" (UID: \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:50:40.245258 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.244114    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-dshm\") pod \"distinct-w4-3fbb1a-kserve-69d65db78b-thc7w\" (UID: \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:50:40.245258 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.244193    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-home\") pod \"distinct-w4-3fbb1a-kserve-69d65db78b-thc7w\" (UID: \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:50:40.245258 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.244245    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-tmp-dir\") pod \"distinct-w4-3fbb1a-kserve-69d65db78b-thc7w\" (UID: \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:50:40.355240 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.349971    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-tls-certs\") pod \"distinct-w4-3fbb1a-kserve-69d65db78b-thc7w\" (UID: \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:50:40.355240 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.350031    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-model-cache\") pod \"distinct-w4-3fbb1a-kserve-69d65db78b-thc7w\" (UID: \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:50:40.355240 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.350060    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfpht\" (UniqueName: \"kubernetes.io/projected/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-kube-api-access-cfpht\") pod \"distinct-w4-3fbb1a-kserve-69d65db78b-thc7w\" (UID: \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:50:40.355240 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.350090    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-dshm\") pod \"distinct-w4-3fbb1a-kserve-69d65db78b-thc7w\" (UID: \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:50:40.355240 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.350182    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-home\") pod \"distinct-w4-3fbb1a-kserve-69d65db78b-thc7w\" (UID: \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:50:40.355240 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.350234    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-tmp-dir\") pod \"distinct-w4-3fbb1a-kserve-69d65db78b-thc7w\" (UID: \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:50:40.355240 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.350872    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-tmp-dir\") pod \"distinct-w4-3fbb1a-kserve-69d65db78b-thc7w\" (UID: \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:50:40.355240 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.351835    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-home\") pod \"distinct-w4-3fbb1a-kserve-69d65db78b-thc7w\" (UID: \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:50:40.355240 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.352263    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-model-cache\") pod \"distinct-w4-3fbb1a-kserve-69d65db78b-thc7w\" (UID: \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:50:40.378244 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.375339    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-tls-certs\") pod \"distinct-w4-3fbb1a-kserve-69d65db78b-thc7w\" (UID: \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:50:40.378417 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.378342    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:50:40.378497 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.378405    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s" podUID="a223486f-1753-4dd1-8fd5-a6c03d664555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 19:50:40.379129 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.379073    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-dshm\") pod \"distinct-w4-3fbb1a-kserve-69d65db78b-thc7w\" (UID: \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:50:40.400791 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.400674    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfpht\" (UniqueName: \"kubernetes.io/projected/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-kube-api-access-cfpht\") pod \"distinct-w4-3fbb1a-kserve-69d65db78b-thc7w\" (UID: \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:50:40.464734 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.464483    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:50:40.639534 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.635624    5591 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:50:40.639534 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.635699    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7" podUID="3c70d023-1700-42d0-964f-5573383d1355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 19:50:40.880238 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:40.880124    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"]
Nov 03 19:50:41.379104 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:41.379067    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 19:50:41.379258 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:41.379144    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s" podUID="a223486f-1753-4dd1-8fd5-a6c03d664555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 19:50:41.629490 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:41.629390    5591 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 19:50:41.629490 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:41.629450    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7" podUID="3c70d023-1700-42d0-964f-5573383d1355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 19:50:41.798486 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:41.798449    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" event={"ID":"dc613fc8-12f6-4cfb-9ab1-3a112eaced51","Type":"ContainerStarted","Data":"87813c228798aea130e8d2e12cac5b59a3533e7b8d7a8453209989e8d6ac0d70"}
Nov 03 19:50:41.806230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:41.806178    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" event={"ID":"dc613fc8-12f6-4cfb-9ab1-3a112eaced51","Type":"ContainerStarted","Data":"1efb3380a78c9bdbbecce00892c93e7fddbbe490132d6aa03c2ec1fd3969c742"}
Nov 03 19:50:41.828241 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:41.828022    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podStartSLOduration=1.8280020750000001 podStartE2EDuration="1.828002075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:41.82662253 +0000 UTC m=+790.598915551" watchObservedRunningTime="2026-11-03 19:50:41.828002075 +0000 UTC m=+790.600295094"
Nov 03 19:50:42.465119 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:42.465083    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:50:42.633721 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:42.633694    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:42.634162 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:42.634141    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:42.634819 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:42.634801    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 19:50:43.420404 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:43.420359    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:50:43.420595 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:43.420434    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s" podUID="a223486f-1753-4dd1-8fd5-a6c03d664555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:50:43.532706 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:43.532666    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:43.533215 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:43.533058    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:43.533923 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:43.533899    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 19:50:44.950022 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:44.949991    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:50:49.099330 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.099289    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"]
Nov 03 19:50:49.103822 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.103798    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.123050 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.123017    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"]
Nov 03 19:50:49.141795 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.141766    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.142268 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.142244    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.142428 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.142411    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.142579 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.142561    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.142687 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.142674    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c9a677a-d699-4b95-b217-09c48c89dff5-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.142824 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.142810    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.143024 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.142981    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mphk8\" (UniqueName: \"kubernetes.io/projected/3c9a677a-d699-4b95-b217-09c48c89dff5-kube-api-access-mphk8\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.143180 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.143059    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.143180 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.143085    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.244734 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.244703    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.244734 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.244741    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c9a677a-d699-4b95-b217-09c48c89dff5-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.244976 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.244790    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.244976 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.244862    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mphk8\" (UniqueName: \"kubernetes.io/projected/3c9a677a-d699-4b95-b217-09c48c89dff5-kube-api-access-mphk8\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.244976 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.244920    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.244976 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.244954    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.245167 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.244993    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.245167 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.245072    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.245167 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.245113    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.245377 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.245165    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.245377 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.245265    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.245377 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.245333    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.245572 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.245550    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.245636 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.245616    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c9a677a-d699-4b95-b217-09c48c89dff5-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.247367 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.247344    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.247581 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.247563    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.261868 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.261843    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.265118 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.265092    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mphk8\" (UniqueName: \"kubernetes.io/projected/3c9a677a-d699-4b95-b217-09c48c89dff5-kube-api-access-mphk8\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.429281 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.424464    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:49.778259 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.778224    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-79986d969f-ch5w8"]
Nov 03 19:50:49.782737 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.782716    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79986d969f-ch5w8"
Nov 03 19:50:49.793974 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.793928    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-745f6b0f9db2\""
Nov 03 19:50:49.794134 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.794060    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-883bbf1ee718\""
Nov 03 19:50:49.797102 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.797077    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-79986d969f-ch5w8"]
Nov 03 19:50:49.862347 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.862322    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-provider-credentials\") pod \"payload-processing-external-model-79986d969f-ch5w8\" (UID: \"a6149f17-bdd7-44fb-a7dc-1314f89a38a6\") " pod="models-as-a-service/payload-processing-external-model-79986d969f-ch5w8"
Nov 03 19:50:49.862420 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.862366    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kwq9\" (UniqueName: \"kubernetes.io/projected/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-kube-api-access-2kwq9\") pod \"payload-processing-external-model-79986d969f-ch5w8\" (UID: \"a6149f17-bdd7-44fb-a7dc-1314f89a38a6\") " pod="models-as-a-service/payload-processing-external-model-79986d969f-ch5w8"
Nov 03 19:50:49.862479 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.862464    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"]
Nov 03 19:50:49.862655 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.862626    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-plugins-config-volume\") pod \"payload-processing-external-model-79986d969f-ch5w8\" (UID: \"a6149f17-bdd7-44fb-a7dc-1314f89a38a6\") " pod="models-as-a-service/payload-processing-external-model-79986d969f-ch5w8"
Nov 03 19:50:49.862718 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.862704    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-routing-overlay-volume\") pod \"payload-processing-external-model-79986d969f-ch5w8\" (UID: \"a6149f17-bdd7-44fb-a7dc-1314f89a38a6\") " pod="models-as-a-service/payload-processing-external-model-79986d969f-ch5w8"
Nov 03 19:50:49.864965 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:50:49.864935    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9a677a_d699_4b95_b217_09c48c89dff5.slice/crio-2ae6c2be53e5d90f394ac30a7bc90ca99908a4740aac0ea3c7a99391a20c2343 WatchSource:0}: Error finding container 2ae6c2be53e5d90f394ac30a7bc90ca99908a4740aac0ea3c7a99391a20c2343: Status 404 returned error can't find the container with id 2ae6c2be53e5d90f394ac30a7bc90ca99908a4740aac0ea3c7a99391a20c2343
Nov 03 19:50:49.867866 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.867834    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:50:49.867975 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.867909    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:50:49.867975 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.867953    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:50:49.963452 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.963418    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-plugins-config-volume\") pod \"payload-processing-external-model-79986d969f-ch5w8\" (UID: \"a6149f17-bdd7-44fb-a7dc-1314f89a38a6\") " pod="models-as-a-service/payload-processing-external-model-79986d969f-ch5w8"
Nov 03 19:50:49.963622 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.963479    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-routing-overlay-volume\") pod \"payload-processing-external-model-79986d969f-ch5w8\" (UID: \"a6149f17-bdd7-44fb-a7dc-1314f89a38a6\") " pod="models-as-a-service/payload-processing-external-model-79986d969f-ch5w8"
Nov 03 19:50:49.963622 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.963543    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-provider-credentials\") pod \"payload-processing-external-model-79986d969f-ch5w8\" (UID: \"a6149f17-bdd7-44fb-a7dc-1314f89a38a6\") " pod="models-as-a-service/payload-processing-external-model-79986d969f-ch5w8"
Nov 03 19:50:49.963622 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.963576    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kwq9\" (UniqueName: \"kubernetes.io/projected/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-kube-api-access-2kwq9\") pod \"payload-processing-external-model-79986d969f-ch5w8\" (UID: \"a6149f17-bdd7-44fb-a7dc-1314f89a38a6\") " pod="models-as-a-service/payload-processing-external-model-79986d969f-ch5w8"
Nov 03 19:50:49.964673 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.964645    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-plugins-config-volume\") pod \"payload-processing-external-model-79986d969f-ch5w8\" (UID: \"a6149f17-bdd7-44fb-a7dc-1314f89a38a6\") " pod="models-as-a-service/payload-processing-external-model-79986d969f-ch5w8"
Nov 03 19:50:49.964918 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.964897    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-routing-overlay-volume\") pod \"payload-processing-external-model-79986d969f-ch5w8\" (UID: \"a6149f17-bdd7-44fb-a7dc-1314f89a38a6\") " pod="models-as-a-service/payload-processing-external-model-79986d969f-ch5w8"
Nov 03 19:50:49.967005 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.966984    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-provider-credentials\") pod \"payload-processing-external-model-79986d969f-ch5w8\" (UID: \"a6149f17-bdd7-44fb-a7dc-1314f89a38a6\") " pod="models-as-a-service/payload-processing-external-model-79986d969f-ch5w8"
Nov 03 19:50:49.976410 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:49.976383    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kwq9\" (UniqueName: \"kubernetes.io/projected/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-kube-api-access-2kwq9\") pod \"payload-processing-external-model-79986d969f-ch5w8\" (UID: \"a6149f17-bdd7-44fb-a7dc-1314f89a38a6\") " pod="models-as-a-service/payload-processing-external-model-79986d969f-ch5w8"
Nov 03 19:50:50.098143 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:50.098055    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79986d969f-ch5w8"
Nov 03 19:50:50.446237 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:50.446191    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-79986d969f-ch5w8"]
Nov 03 19:50:50.448359 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:50:50.448331    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6149f17_bdd7_44fb_a7dc_1314f89a38a6.slice/crio-04adcea4f396644b76d0de9dca26fcc939650d020ef51a5e1e34bde31ee163df WatchSource:0}: Error finding container 04adcea4f396644b76d0de9dca26fcc939650d020ef51a5e1e34bde31ee163df: Status 404 returned error can't find the container with id 04adcea4f396644b76d0de9dca26fcc939650d020ef51a5e1e34bde31ee163df
Nov 03 19:50:50.465162 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:50.465142    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:50:50.503299 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:50.496174    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:50:50.862112 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:50.862076    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq" event={"ID":"3c9a677a-d699-4b95-b217-09c48c89dff5","Type":"ContainerStarted","Data":"53f6267c02996dfe84da999d23e8a68e263db947788f32841483738cff87eddd"}
Nov 03 19:50:50.862343 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:50.862155    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq" event={"ID":"3c9a677a-d699-4b95-b217-09c48c89dff5","Type":"ContainerStarted","Data":"2ae6c2be53e5d90f394ac30a7bc90ca99908a4740aac0ea3c7a99391a20c2343"}
Nov 03 19:50:50.863925 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:50.863897    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79986d969f-ch5w8" event={"ID":"a6149f17-bdd7-44fb-a7dc-1314f89a38a6","Type":"ContainerStarted","Data":"5453b0e62130643b6496895f5c5e0589c6930eb5f50000344ca55de9137c62de"}
Nov 03 19:50:50.864048 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:50.863945    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79986d969f-ch5w8" event={"ID":"a6149f17-bdd7-44fb-a7dc-1314f89a38a6","Type":"ContainerStarted","Data":"04adcea4f396644b76d0de9dca26fcc939650d020ef51a5e1e34bde31ee163df"}
Nov 03 19:50:50.864123 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:50.864108    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-79986d969f-ch5w8"
Nov 03 19:50:50.875944 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:50.875920    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:50:50.901758 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:50.901691    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq" podStartSLOduration=1.901670502 podStartE2EDuration="1.901670502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:50.901660154 +0000 UTC m=+799.673953175" watchObservedRunningTime="2026-11-03 19:50:50.901670502 +0000 UTC m=+799.673963522"
Nov 03 19:50:50.926346 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:50.926284    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-79986d969f-ch5w8" podStartSLOduration=1.9262637150000002 podStartE2EDuration="1.926263715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:50.926031913 +0000 UTC m=+799.698324931" watchObservedRunningTime="2026-11-03 19:50:50.926263715 +0000 UTC m=+799.698556764"
Nov 03 19:50:51.430639 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:51.430603    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:51.432548 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:51.432520    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 19:50:51.432680 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:51.432574    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq" podUID="3c9a677a-d699-4b95-b217-09c48c89dff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 19:50:52.431061 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:52.431016    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 19:50:52.431524 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:52.431088    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq" podUID="3c9a677a-d699-4b95-b217-09c48c89dff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 19:50:53.430688 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:53.430648    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 19:50:53.430864 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:53.430712    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq" podUID="3c9a677a-d699-4b95-b217-09c48c89dff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 19:50:54.431219 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:54.431163    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 19:50:54.431716 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:54.431243    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq" podUID="3c9a677a-d699-4b95-b217-09c48c89dff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 19:50:55.430816 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:55.430783    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 19:50:55.431026 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:55.430844    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq" podUID="3c9a677a-d699-4b95-b217-09c48c89dff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 19:50:56.430896 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:56.430861    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 03 19:50:56.431324 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:56.430923    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq" podUID="3c9a677a-d699-4b95-b217-09c48c89dff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 03 19:50:57.435487 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:57.435455    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:57.435898 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:57.435712    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:57.436381 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:50:57.436366    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 19:50:58.902755 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:50:58.902686    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:51:01.877105 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:01.877071    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-79986d969f-ch5w8"
Nov 03 19:51:01.947970 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:01.947937    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t"]
Nov 03 19:51:04.346096 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.346063    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"]
Nov 03 19:51:04.352156 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.352132    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.366962 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.366938    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"]
Nov 03 19:51:04.443480 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.443447    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.443651 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.443501    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.443651 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.443580    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.443732 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.443653    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.443732 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.443686    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.443732 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.443713    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxqz2\" (UniqueName: \"kubernetes.io/projected/b043886c-68e1-48d8-9ec4-8ecd4601a753-kube-api-access-lxqz2\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.443848 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.443832    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.443920 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.443908    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.443955 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.443932    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b043886c-68e1-48d8-9ec4-8ecd4601a753-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.544676 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.544633    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.544676 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.544678    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b043886c-68e1-48d8-9ec4-8ecd4601a753-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.544938 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.544711    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.544938 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.544748    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.544938 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.544789    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.544938 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.544829    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.544938 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.544855    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.544938 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.544879    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxqz2\" (UniqueName: \"kubernetes.io/projected/b043886c-68e1-48d8-9ec4-8ecd4601a753-kube-api-access-lxqz2\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.545247 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.544954    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.545247 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.545108    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-workload-certs\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.545339 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.545280    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-credential-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.545384 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.545350    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-workload-socket\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.545475 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.545447    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-data\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.545835 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.545810    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b043886c-68e1-48d8-9ec4-8ecd4601a753-istiod-ca-cert\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.547578 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.547554    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-envoy\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.547690 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.547652    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-podinfo\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.557407 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.557370    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-token\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.557664 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.557640    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxqz2\" (UniqueName: \"kubernetes.io/projected/b043886c-68e1-48d8-9ec4-8ecd4601a753-kube-api-access-lxqz2\") pod \"e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") " pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.668341 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.668314    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:04.685742 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.685707    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:51:04.983352 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.983322    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"]
Nov 03 19:51:04.984988 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:51:04.984959    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb043886c_68e1_48d8_9ec4_8ecd4601a753.slice/crio-2c8b6abb912158a274652b24b8b8ed0477ac587c6bd1aba78afcfd193ac2108f WatchSource:0}: Error finding container 2c8b6abb912158a274652b24b8b8ed0477ac587c6bd1aba78afcfd193ac2108f: Status 404 returned error can't find the container with id 2c8b6abb912158a274652b24b8b8ed0477ac587c6bd1aba78afcfd193ac2108f
Nov 03 19:51:04.987438 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.987406    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:51:04.987515 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.987470    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:51:04.987515 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:04.987499    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:51:05.117430 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:05.117397    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr" event={"ID":"b043886c-68e1-48d8-9ec4-8ecd4601a753","Type":"ContainerStarted","Data":"7caaeec4c314cef19e22813fb950cf5661b3f97fe0c26e48513fc643739a1b62"}
Nov 03 19:51:05.117617 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:05.117454    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr" event={"ID":"b043886c-68e1-48d8-9ec4-8ecd4601a753","Type":"ContainerStarted","Data":"2c8b6abb912158a274652b24b8b8ed0477ac587c6bd1aba78afcfd193ac2108f"}
Nov 03 19:51:05.142403 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:05.142156    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr" podStartSLOduration=1.14214132 podStartE2EDuration="1.14214132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:05.141397573 +0000 UTC m=+813.913690591" watchObservedRunningTime="2026-11-03 19:51:05.14214132 +0000 UTC m=+813.914434401"
Nov 03 19:51:05.683244 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:05.670315    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:06.435226 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:06.431950    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"]
Nov 03 19:51:06.519504 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:06.492394    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"]
Nov 03 19:51:06.519504 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:06.492517    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:51:06.532243 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:06.531004    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"distinct-2-w4-3fbb1a-kserve-self-signed-certs\""
Nov 03 19:51:06.614547 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:06.614504    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-home\") pod \"distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x\" (UID: \"2c1b25f8-ea79-44e9-991f-1067e71cf960\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:51:06.698286 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:06.697722    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-dshm\") pod \"distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x\" (UID: \"2c1b25f8-ea79-44e9-991f-1067e71cf960\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:51:06.698286 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:06.697787    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c1b25f8-ea79-44e9-991f-1067e71cf960-tls-certs\") pod \"distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x\" (UID: \"2c1b25f8-ea79-44e9-991f-1067e71cf960\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:51:06.698286 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:06.697821    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhlkk\" (UniqueName: \"kubernetes.io/projected/2c1b25f8-ea79-44e9-991f-1067e71cf960-kube-api-access-mhlkk\") pod \"distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x\" (UID: \"2c1b25f8-ea79-44e9-991f-1067e71cf960\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:51:06.698286 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:06.697869    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-model-cache\") pod \"distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x\" (UID: \"2c1b25f8-ea79-44e9-991f-1067e71cf960\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:51:06.698286 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:06.697954    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-tmp-dir\") pod \"distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x\" (UID: \"2c1b25f8-ea79-44e9-991f-1067e71cf960\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:51:06.826561 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:06.804516    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-tmp-dir\") pod \"distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x\" (UID: \"2c1b25f8-ea79-44e9-991f-1067e71cf960\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:51:06.826561 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:06.804676    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-home\") pod \"distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x\" (UID: \"2c1b25f8-ea79-44e9-991f-1067e71cf960\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:51:06.826561 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:06.804757    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-dshm\") pod \"distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x\" (UID: \"2c1b25f8-ea79-44e9-991f-1067e71cf960\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:51:06.826561 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:06.804801    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c1b25f8-ea79-44e9-991f-1067e71cf960-tls-certs\") pod \"distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x\" (UID: \"2c1b25f8-ea79-44e9-991f-1067e71cf960\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:51:06.826561 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:06.804827    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhlkk\" (UniqueName: \"kubernetes.io/projected/2c1b25f8-ea79-44e9-991f-1067e71cf960-kube-api-access-mhlkk\") pod \"distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x\" (UID: \"2c1b25f8-ea79-44e9-991f-1067e71cf960\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:51:06.826561 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:06.804866    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-model-cache\") pod \"distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x\" (UID: \"2c1b25f8-ea79-44e9-991f-1067e71cf960\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:51:06.855366 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:06.829550    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-home\") pod \"distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x\" (UID: \"2c1b25f8-ea79-44e9-991f-1067e71cf960\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:51:06.855366 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:06.829840    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-tmp-dir\") pod \"distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x\" (UID: \"2c1b25f8-ea79-44e9-991f-1067e71cf960\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:51:06.857684 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:06.857656    5591 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 19:51:06.857899 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:06.857872    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr" podUID="b043886c-68e1-48d8-9ec4-8ecd4601a753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 19:51:06.925591 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:06.925506    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-model-cache\") pod \"distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x\" (UID: \"2c1b25f8-ea79-44e9-991f-1067e71cf960\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:51:06.948832 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:06.947540    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-dshm\") pod \"distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x\" (UID: \"2c1b25f8-ea79-44e9-991f-1067e71cf960\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:51:06.974495 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:06.972907    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c1b25f8-ea79-44e9-991f-1067e71cf960-tls-certs\") pod \"distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x\" (UID: \"2c1b25f8-ea79-44e9-991f-1067e71cf960\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:51:06.974495 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:06.973937    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t" podUID="2168bf74-6936-46ea-b16a-d73af51825ee" containerName="payload-processing" containerID="cri-o://adeeedc9ba24f92a81219b95c794e7dac11d5337fd97aad809a9463b90a8804c" gracePeriod=25
Nov 03 19:51:07.064049 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:07.044878    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhlkk\" (UniqueName: \"kubernetes.io/projected/2c1b25f8-ea79-44e9-991f-1067e71cf960-kube-api-access-mhlkk\") pod \"distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x\" (UID: \"2c1b25f8-ea79-44e9-991f-1067e71cf960\") " pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:51:07.064049 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:07.062646    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:51:07.686232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:07.674573    5591 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 19:51:07.686232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:07.674632    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr" podUID="b043886c-68e1-48d8-9ec4-8ecd4601a753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 19:51:08.086936 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:08.073321    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 19:51:08.468517 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:08.453518    5591 generic.go:356] "Generic (PLEG): container finished" podID="2168bf74-6936-46ea-b16a-d73af51825ee" containerID="adeeedc9ba24f92a81219b95c794e7dac11d5337fd97aad809a9463b90a8804c" exitCode=0
Nov 03 19:51:08.468517 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:08.453751    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t" event={"ID":"2168bf74-6936-46ea-b16a-d73af51825ee","Type":"ContainerDied","Data":"adeeedc9ba24f92a81219b95c794e7dac11d5337fd97aad809a9463b90a8804c"}
Nov 03 19:51:08.584264 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:08.575919    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"]
Nov 03 19:51:08.717309 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:08.714389    5591 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 19:51:08.717309 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:08.714452    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr" podUID="b043886c-68e1-48d8-9ec4-8ecd4601a753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 19:51:09.016360 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:09.013048    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t"
Nov 03 19:51:09.179231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:09.176687    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2168bf74-6936-46ea-b16a-d73af51825ee-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2168bf74-6936-46ea-b16a-d73af51825ee-routing-overlay-volume\") pod \"2168bf74-6936-46ea-b16a-d73af51825ee\" (UID: \"2168bf74-6936-46ea-b16a-d73af51825ee\") "
Nov 03 19:51:09.179231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:09.176820    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2168bf74-6936-46ea-b16a-d73af51825ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2168bf74-6936-46ea-b16a-d73af51825ee-plugins-config-volume\") pod \"2168bf74-6936-46ea-b16a-d73af51825ee\" (UID: \"2168bf74-6936-46ea-b16a-d73af51825ee\") "
Nov 03 19:51:09.179231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:09.176914    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2168bf74-6936-46ea-b16a-d73af51825ee-kube-api-access-m4v9c\" (UniqueName: \"kubernetes.io/projected/2168bf74-6936-46ea-b16a-d73af51825ee-kube-api-access-m4v9c\") pod \"2168bf74-6936-46ea-b16a-d73af51825ee\" (UID: \"2168bf74-6936-46ea-b16a-d73af51825ee\") "
Nov 03 19:51:09.179231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:09.177019    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2168bf74-6936-46ea-b16a-d73af51825ee-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2168bf74-6936-46ea-b16a-d73af51825ee-provider-credentials\") pod \"2168bf74-6936-46ea-b16a-d73af51825ee\" (UID: \"2168bf74-6936-46ea-b16a-d73af51825ee\") "
Nov 03 19:51:09.221259 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:09.212422    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2168bf74-6936-46ea-b16a-d73af51825ee-plugins-config-volume" pod "2168bf74-6936-46ea-b16a-d73af51825ee" (UID: "2168bf74-6936-46ea-b16a-d73af51825ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:51:09.317310 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:09.306506    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2168bf74-6936-46ea-b16a-d73af51825ee-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:09.317310 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:09.307385    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2168bf74-6936-46ea-b16a-d73af51825ee-provider-credentials" pod "2168bf74-6936-46ea-b16a-d73af51825ee" (UID: "2168bf74-6936-46ea-b16a-d73af51825ee"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:09.317310 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:09.309334    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2168bf74-6936-46ea-b16a-d73af51825ee-kube-api-access-m4v9c" pod "2168bf74-6936-46ea-b16a-d73af51825ee" (UID: "2168bf74-6936-46ea-b16a-d73af51825ee"). InnerVolumeSpecName "kube-api-access-m4v9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:09.317310 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:09.233330    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2168bf74-6936-46ea-b16a-d73af51825ee-routing-overlay-volume" pod "2168bf74-6936-46ea-b16a-d73af51825ee" (UID: "2168bf74-6936-46ea-b16a-d73af51825ee"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:51:09.408230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:09.408137    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4v9c\" (UniqueName: \"kubernetes.io/projected/2168bf74-6936-46ea-b16a-d73af51825ee-kube-api-access-m4v9c\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:09.408230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:09.408175    5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2168bf74-6936-46ea-b16a-d73af51825ee-provider-credentials\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:09.408230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:09.408190    5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2168bf74-6936-46ea-b16a-d73af51825ee-routing-overlay-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:09.543231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:09.542656    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t" event={"ID":"2168bf74-6936-46ea-b16a-d73af51825ee","Type":"ContainerDied","Data":"2358f7e85b1b5af5f740be9efbb0ed2d7e60069c459aa1d561310d0b0a2c1f69"}
Nov 03 19:51:09.543231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:09.542733    5591 scope.go:122] "RemoveContainer" containerID="adeeedc9ba24f92a81219b95c794e7dac11d5337fd97aad809a9463b90a8804c"
Nov 03 19:51:09.543231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:09.542912    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t"
Nov 03 19:51:09.595064 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:09.594904    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" event={"ID":"2c1b25f8-ea79-44e9-991f-1067e71cf960","Type":"ContainerStarted","Data":"f635f6627a4271c66ca3a103d7e5622ca157d54bb44ce0239265f9a75bca3c99"}
Nov 03 19:51:09.693821 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:09.686872    5591 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 19:51:09.693821 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:09.686919    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr" podUID="b043886c-68e1-48d8-9ec4-8ecd4601a753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 19:51:09.697922 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:09.676188    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t"]
Nov 03 19:51:09.698223 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:09.698174    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-65d589b9db-w6g9t"]
Nov 03 19:51:09.913452 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:09.913347    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:10.559232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:10.558962    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:10.689229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:10.683587    5591 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 19:51:10.689229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:10.683654    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr" podUID="b043886c-68e1-48d8-9ec4-8ecd4601a753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 19:51:10.689229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:10.688285    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:10.724099 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:10.724056    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" event={"ID":"2c1b25f8-ea79-44e9-991f-1067e71cf960","Type":"ContainerStarted","Data":"920db3ea2196aacbe6905c33c7084feccf0c5a5d2e1db64577815bcf2c60e1ab"}
Nov 03 19:51:10.767840 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:10.767774    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podStartSLOduration=4.767752716 podStartE2EDuration="4.767752716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:10.764631222 +0000 UTC m=+819.536924237" watchObservedRunningTime="2026-11-03 19:51:10.767752716 +0000 UTC m=+819.540045735"
Nov 03 19:51:10.929293 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:10.923304    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 19:51:10.929293 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:10.924253    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:51:11.038055 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:11.036815    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:51:11.064573 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:11.064543    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:51:11.516283 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:11.501264    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2168bf74-6936-46ea-b16a-d73af51825ee" path="/var/lib/kubelet/pods/2168bf74-6936-46ea-b16a-d73af51825ee/volumes"
Nov 03 19:51:11.671233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:11.668886    5591 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 19:51:11.671233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:11.668964    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr" podUID="b043886c-68e1-48d8-9ec4-8ecd4601a753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 19:51:12.669614 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:12.669573    5591 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 19:51:12.669783 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:12.669643    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr" podUID="b043886c-68e1-48d8-9ec4-8ecd4601a753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 19:51:13.668705 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:13.668672    5591 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 19:51:13.669127 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:13.668723    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr" podUID="b043886c-68e1-48d8-9ec4-8ecd4601a753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 19:51:14.673243 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:14.673191    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:14.673739 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:14.673715    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:14.674582 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:14.674561    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 19:51:17.075234 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:17.071429    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:51:17.195223 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:17.195175    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:51:18.185230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.184186    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:51:18.251230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.250254    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"]
Nov 03 19:51:18.251230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.251128    5591 cpu_manager.go:414] "containerMap: removing container" podUID="2168bf74-6936-46ea-b16a-d73af51825ee" containerName="payload-processing"
Nov 03 19:51:18.251230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.251149    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2168bf74-6936-46ea-b16a-d73af51825ee" containerName="payload-processing"
Nov 03 19:51:18.251536 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.251352    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="2168bf74-6936-46ea-b16a-d73af51825ee" containerName="payload-processing"
Nov 03 19:51:18.306229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.305589    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"]
Nov 03 19:51:18.306229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.305714    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:51:18.316228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.311374    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"distinct-2-w1-32e482-kserve-self-signed-certs\""
Nov 03 19:51:18.316228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.311563    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"openshift-service-ca.crt\""
Nov 03 19:51:18.316228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.311652    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"kube-root-ca.crt\""
Nov 03 19:51:18.316228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.311745    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"default-dockercfg-5r8nn\""
Nov 03 19:51:18.412242 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.412076    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-dshm\") pod \"distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2\" (UID: \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:51:18.412242 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.412152    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-tmp-dir\") pod \"distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2\" (UID: \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:51:18.412499 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.412271    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-home\") pod \"distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2\" (UID: \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:51:18.412499 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.412386    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-tls-certs\") pod \"distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2\" (UID: \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:51:18.412499 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.412422    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-model-cache\") pod \"distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2\" (UID: \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:51:18.412499 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.412493    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpwsw\" (UniqueName: \"kubernetes.io/projected/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-kube-api-access-jpwsw\") pod \"distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2\" (UID: \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:51:18.632287 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.515081    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-tls-certs\") pod \"distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2\" (UID: \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:51:18.632287 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.515138    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-model-cache\") pod \"distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2\" (UID: \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:51:18.632287 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.515267    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpwsw\" (UniqueName: \"kubernetes.io/projected/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-kube-api-access-jpwsw\") pod \"distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2\" (UID: \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:51:18.632287 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.515381    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-dshm\") pod \"distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2\" (UID: \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:51:18.632287 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.515428    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-tmp-dir\") pod \"distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2\" (UID: \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:51:18.632287 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.515501    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-home\") pod \"distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2\" (UID: \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:51:18.632287 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.516142    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-home\") pod \"distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2\" (UID: \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:51:18.632287 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.541113    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-tmp-dir\") pod \"distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2\" (UID: \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:51:18.632287 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.547381    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-model-cache\") pod \"distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2\" (UID: \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:51:18.632287 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.587338    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-tls-certs\") pod \"distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2\" (UID: \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:51:18.648765 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.648509    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-dshm\") pod \"distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2\" (UID: \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:51:18.798230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.675786    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpwsw\" (UniqueName: \"kubernetes.io/projected/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-kube-api-access-jpwsw\") pod \"distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2\" (UID: \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\") " pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:51:18.803285 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:18.800804    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:51:19.179195 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:19.179170    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"]
Nov 03 19:51:19.180832 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:51:19.180807    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda240d836_cfa9_48a2_9bb0_ad49ed9da4d0.slice/crio-495b2d71efb3a69b016f26696396a72e849667a9c6a1e4444c17e7f7184af2a3 WatchSource:0}: Error finding container 495b2d71efb3a69b016f26696396a72e849667a9c6a1e4444c17e7f7184af2a3: Status 404 returned error can't find the container with id 495b2d71efb3a69b016f26696396a72e849667a9c6a1e4444c17e7f7184af2a3
Nov 03 19:51:19.991445 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:19.991413    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" event={"ID":"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0","Type":"ContainerStarted","Data":"cf92c48529a2152532f83b4375a75e7a963ef5d1cd7ed9e0ce9ac5bc8b75ec76"}
Nov 03 19:51:19.991880 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:19.991460    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" event={"ID":"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0","Type":"ContainerStarted","Data":"495b2d71efb3a69b016f26696396a72e849667a9c6a1e4444c17e7f7184af2a3"}
Nov 03 19:51:20.011544 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:20.011505    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podStartSLOduration=2.01149164 podStartE2EDuration="2.01149164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:20.009983317 +0000 UTC m=+828.782276335" watchObservedRunningTime="2026-11-03 19:51:20.01149164 +0000 UTC m=+828.783784657"
Nov 03 19:51:20.804286 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:20.804250    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:51:23.437665 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:23.437626    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:51:26.766642 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:26.764255    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-65d589b9db-prrwd"]
Nov 03 19:51:26.793117 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:26.793090    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65d589b9db-prrwd"
Nov 03 19:51:26.827235 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:26.815136    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-65d589b9db-prrwd"]
Nov 03 19:51:26.918225 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:26.912683    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4167c3-f632-477a-bc97-019193a3a692-plugins-config-volume\") pod \"payload-processing-external-model-65d589b9db-prrwd\" (UID: \"4e4167c3-f632-477a-bc97-019193a3a692\") " pod="models-as-a-service/payload-processing-external-model-65d589b9db-prrwd"
Nov 03 19:51:26.918225 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:26.912748    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwxdj\" (UniqueName: \"kubernetes.io/projected/4e4167c3-f632-477a-bc97-019193a3a692-kube-api-access-fwxdj\") pod \"payload-processing-external-model-65d589b9db-prrwd\" (UID: \"4e4167c3-f632-477a-bc97-019193a3a692\") " pod="models-as-a-service/payload-processing-external-model-65d589b9db-prrwd"
Nov 03 19:51:26.918225 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:26.917386    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e4167c3-f632-477a-bc97-019193a3a692-provider-credentials\") pod \"payload-processing-external-model-65d589b9db-prrwd\" (UID: \"4e4167c3-f632-477a-bc97-019193a3a692\") " pod="models-as-a-service/payload-processing-external-model-65d589b9db-prrwd"
Nov 03 19:51:26.944604 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:26.944472    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e4167c3-f632-477a-bc97-019193a3a692-routing-overlay-volume\") pod \"payload-processing-external-model-65d589b9db-prrwd\" (UID: \"4e4167c3-f632-477a-bc97-019193a3a692\") " pod="models-as-a-service/payload-processing-external-model-65d589b9db-prrwd"
Nov 03 19:51:27.045668 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:27.045564    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e4167c3-f632-477a-bc97-019193a3a692-provider-credentials\") pod \"payload-processing-external-model-65d589b9db-prrwd\" (UID: \"4e4167c3-f632-477a-bc97-019193a3a692\") " pod="models-as-a-service/payload-processing-external-model-65d589b9db-prrwd"
Nov 03 19:51:27.045853 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:27.045764    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e4167c3-f632-477a-bc97-019193a3a692-routing-overlay-volume\") pod \"payload-processing-external-model-65d589b9db-prrwd\" (UID: \"4e4167c3-f632-477a-bc97-019193a3a692\") " pod="models-as-a-service/payload-processing-external-model-65d589b9db-prrwd"
Nov 03 19:51:27.045853 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:27.045833    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4167c3-f632-477a-bc97-019193a3a692-plugins-config-volume\") pod \"payload-processing-external-model-65d589b9db-prrwd\" (UID: \"4e4167c3-f632-477a-bc97-019193a3a692\") " pod="models-as-a-service/payload-processing-external-model-65d589b9db-prrwd"
Nov 03 19:51:27.045965 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:27.045870    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwxdj\" (UniqueName: \"kubernetes.io/projected/4e4167c3-f632-477a-bc97-019193a3a692-kube-api-access-fwxdj\") pod \"payload-processing-external-model-65d589b9db-prrwd\" (UID: \"4e4167c3-f632-477a-bc97-019193a3a692\") " pod="models-as-a-service/payload-processing-external-model-65d589b9db-prrwd"
Nov 03 19:51:27.046843 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:27.046809    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e4167c3-f632-477a-bc97-019193a3a692-routing-overlay-volume\") pod \"payload-processing-external-model-65d589b9db-prrwd\" (UID: \"4e4167c3-f632-477a-bc97-019193a3a692\") " pod="models-as-a-service/payload-processing-external-model-65d589b9db-prrwd"
Nov 03 19:51:27.046965 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:27.046848    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4167c3-f632-477a-bc97-019193a3a692-plugins-config-volume\") pod \"payload-processing-external-model-65d589b9db-prrwd\" (UID: \"4e4167c3-f632-477a-bc97-019193a3a692\") " pod="models-as-a-service/payload-processing-external-model-65d589b9db-prrwd"
Nov 03 19:51:27.049141 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:27.049088    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e4167c3-f632-477a-bc97-019193a3a692-provider-credentials\") pod \"payload-processing-external-model-65d589b9db-prrwd\" (UID: \"4e4167c3-f632-477a-bc97-019193a3a692\") " pod="models-as-a-service/payload-processing-external-model-65d589b9db-prrwd"
Nov 03 19:51:27.066238 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:27.065623    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwxdj\" (UniqueName: \"kubernetes.io/projected/4e4167c3-f632-477a-bc97-019193a3a692-kube-api-access-fwxdj\") pod \"payload-processing-external-model-65d589b9db-prrwd\" (UID: \"4e4167c3-f632-477a-bc97-019193a3a692\") " pod="models-as-a-service/payload-processing-external-model-65d589b9db-prrwd"
Nov 03 19:51:27.256224 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:27.255676    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65d589b9db-prrwd"
Nov 03 19:51:27.596455 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:27.596432    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-65d589b9db-prrwd"]
Nov 03 19:51:27.598105 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:51:27.598061    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4167c3_f632_477a_bc97_019193a3a692.slice/crio-21abad3a149991956b92bbe224fdf3b206efbb83746c77305dbdb734cfb5f3a2 WatchSource:0}: Error finding container 21abad3a149991956b92bbe224fdf3b206efbb83746c77305dbdb734cfb5f3a2: Status 404 returned error can't find the container with id 21abad3a149991956b92bbe224fdf3b206efbb83746c77305dbdb734cfb5f3a2
Nov 03 19:51:28.062496 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:28.062461    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65d589b9db-prrwd" event={"ID":"4e4167c3-f632-477a-bc97-019193a3a692","Type":"ContainerStarted","Data":"637895c1b3d99cedd7197a4cd971bfd91d1c63cb2ce0b8f260e4c1944504fa13"}
Nov 03 19:51:28.062936 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:28.062518    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65d589b9db-prrwd" event={"ID":"4e4167c3-f632-477a-bc97-019193a3a692","Type":"ContainerStarted","Data":"21abad3a149991956b92bbe224fdf3b206efbb83746c77305dbdb734cfb5f3a2"}
Nov 03 19:51:28.062936 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:28.062539    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-65d589b9db-prrwd"
Nov 03 19:51:28.083086 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:28.083030    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-65d589b9db-prrwd" podStartSLOduration=2.0830151040000002 podStartE2EDuration="2.083015104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:28.081840314 +0000 UTC m=+836.854133331" watchObservedRunningTime="2026-11-03 19:51:28.083015104 +0000 UTC m=+836.855308141"
Nov 03 19:51:28.723644 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:51:28.723599    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:51:28.804289 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:28.804259    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:51:28.817675 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:28.817650    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:51:29.081129 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:29.081101    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:51:38.552468 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:38.552434    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-79986d969f-ch5w8"]
Nov 03 19:51:38.555185 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:38.555154    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-65d589b9db-prrwd"]
Nov 03 19:51:38.600384 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:38.600354    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-65d589b9db-prrwd"
Nov 03 19:51:43.553516 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:43.553465    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-79986d969f-ch5w8" podUID="a6149f17-bdd7-44fb-a7dc-1314f89a38a6" containerName="payload-processing" containerID="cri-o://5453b0e62130643b6496895f5c5e0589c6930eb5f50000344ca55de9137c62de" gracePeriod=25
Nov 03 19:51:43.556390 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:43.556350    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-65d589b9db-prrwd" podUID="4e4167c3-f632-477a-bc97-019193a3a692" containerName="payload-processing" containerID="cri-o://637895c1b3d99cedd7197a4cd971bfd91d1c63cb2ce0b8f260e4c1944504fa13" gracePeriod=25
Nov 03 19:51:43.858528 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:43.858494    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79986d969f-ch5w8"
Nov 03 19:51:43.935554 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:43.934890    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65d589b9db-prrwd"
Nov 03 19:51:44.004688 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.004651    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-routing-overlay-volume\") pod \"a6149f17-bdd7-44fb-a7dc-1314f89a38a6\" (UID: \"a6149f17-bdd7-44fb-a7dc-1314f89a38a6\") "
Nov 03 19:51:44.004877 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.004719    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-plugins-config-volume\") pod \"a6149f17-bdd7-44fb-a7dc-1314f89a38a6\" (UID: \"a6149f17-bdd7-44fb-a7dc-1314f89a38a6\") "
Nov 03 19:51:44.004941 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.004878    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-provider-credentials\") pod \"a6149f17-bdd7-44fb-a7dc-1314f89a38a6\" (UID: \"a6149f17-bdd7-44fb-a7dc-1314f89a38a6\") "
Nov 03 19:51:44.004941 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.004931    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-kube-api-access-2kwq9\" (UniqueName: \"kubernetes.io/projected/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-kube-api-access-2kwq9\") pod \"a6149f17-bdd7-44fb-a7dc-1314f89a38a6\" (UID: \"a6149f17-bdd7-44fb-a7dc-1314f89a38a6\") "
Nov 03 19:51:44.005188 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.005154    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-routing-overlay-volume" pod "a6149f17-bdd7-44fb-a7dc-1314f89a38a6" (UID: "a6149f17-bdd7-44fb-a7dc-1314f89a38a6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:51:44.005343 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.005320    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-plugins-config-volume" pod "a6149f17-bdd7-44fb-a7dc-1314f89a38a6" (UID: "a6149f17-bdd7-44fb-a7dc-1314f89a38a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:51:44.005872 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.005825    5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-routing-overlay-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:44.005872 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.005855    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:44.007502 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.007469    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-kube-api-access-2kwq9" pod "a6149f17-bdd7-44fb-a7dc-1314f89a38a6" (UID: "a6149f17-bdd7-44fb-a7dc-1314f89a38a6"). InnerVolumeSpecName "kube-api-access-2kwq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:44.007677 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.007658    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-provider-credentials" pod "a6149f17-bdd7-44fb-a7dc-1314f89a38a6" (UID: "a6149f17-bdd7-44fb-a7dc-1314f89a38a6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:44.107003 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.106915    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4167c3-f632-477a-bc97-019193a3a692-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e4167c3-f632-477a-bc97-019193a3a692-provider-credentials\") pod \"4e4167c3-f632-477a-bc97-019193a3a692\" (UID: \"4e4167c3-f632-477a-bc97-019193a3a692\") "
Nov 03 19:51:44.107192 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.107006    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e4167c3-f632-477a-bc97-019193a3a692-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e4167c3-f632-477a-bc97-019193a3a692-routing-overlay-volume\") pod \"4e4167c3-f632-477a-bc97-019193a3a692\" (UID: \"4e4167c3-f632-477a-bc97-019193a3a692\") "
Nov 03 19:51:44.107192 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.107114    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4167c3-f632-477a-bc97-019193a3a692-kube-api-access-fwxdj\" (UniqueName: \"kubernetes.io/projected/4e4167c3-f632-477a-bc97-019193a3a692-kube-api-access-fwxdj\") pod \"4e4167c3-f632-477a-bc97-019193a3a692\" (UID: \"4e4167c3-f632-477a-bc97-019193a3a692\") "
Nov 03 19:51:44.107351 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.107252    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e4167c3-f632-477a-bc97-019193a3a692-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4167c3-f632-477a-bc97-019193a3a692-plugins-config-volume\") pod \"4e4167c3-f632-477a-bc97-019193a3a692\" (UID: \"4e4167c3-f632-477a-bc97-019193a3a692\") "
Nov 03 19:51:44.107568 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.107541    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4167c3-f632-477a-bc97-019193a3a692-routing-overlay-volume" pod "4e4167c3-f632-477a-bc97-019193a3a692" (UID: "4e4167c3-f632-477a-bc97-019193a3a692"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:51:44.107834 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.107815    5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4e4167c3-f632-477a-bc97-019193a3a692-routing-overlay-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:44.108109 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.107836    5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-provider-credentials\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:44.108109 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.107847    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kwq9\" (UniqueName: \"kubernetes.io/projected/a6149f17-bdd7-44fb-a7dc-1314f89a38a6-kube-api-access-2kwq9\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:44.108109 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.107829    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4167c3-f632-477a-bc97-019193a3a692-plugins-config-volume" pod "4e4167c3-f632-477a-bc97-019193a3a692" (UID: "4e4167c3-f632-477a-bc97-019193a3a692"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:51:44.109415 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.109370    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4167c3-f632-477a-bc97-019193a3a692-provider-credentials" pod "4e4167c3-f632-477a-bc97-019193a3a692" (UID: "4e4167c3-f632-477a-bc97-019193a3a692"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:44.109617 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.109593    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4167c3-f632-477a-bc97-019193a3a692-kube-api-access-fwxdj" pod "4e4167c3-f632-477a-bc97-019193a3a692" (UID: "4e4167c3-f632-477a-bc97-019193a3a692"). InnerVolumeSpecName "kube-api-access-fwxdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:44.209088 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.209054    5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4e4167c3-f632-477a-bc97-019193a3a692-provider-credentials\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:44.209088 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.209081    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwxdj\" (UniqueName: \"kubernetes.io/projected/4e4167c3-f632-477a-bc97-019193a3a692-kube-api-access-fwxdj\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:44.209088 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.209092    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4167c3-f632-477a-bc97-019193a3a692-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:44.223983 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.223944    5591 generic.go:356] "Generic (PLEG): container finished" podID="a6149f17-bdd7-44fb-a7dc-1314f89a38a6" containerID="5453b0e62130643b6496895f5c5e0589c6930eb5f50000344ca55de9137c62de" exitCode=0
Nov 03 19:51:44.224181 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.223998    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79986d969f-ch5w8"
Nov 03 19:51:44.224181 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.224019    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79986d969f-ch5w8" event={"ID":"a6149f17-bdd7-44fb-a7dc-1314f89a38a6","Type":"ContainerDied","Data":"5453b0e62130643b6496895f5c5e0589c6930eb5f50000344ca55de9137c62de"}
Nov 03 19:51:44.224181 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.224107    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79986d969f-ch5w8" event={"ID":"a6149f17-bdd7-44fb-a7dc-1314f89a38a6","Type":"ContainerDied","Data":"04adcea4f396644b76d0de9dca26fcc939650d020ef51a5e1e34bde31ee163df"}
Nov 03 19:51:44.224181 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.224136    5591 scope.go:122] "RemoveContainer" containerID="5453b0e62130643b6496895f5c5e0589c6930eb5f50000344ca55de9137c62de"
Nov 03 19:51:44.225553 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.225527    5591 generic.go:356] "Generic (PLEG): container finished" podID="4e4167c3-f632-477a-bc97-019193a3a692" containerID="637895c1b3d99cedd7197a4cd971bfd91d1c63cb2ce0b8f260e4c1944504fa13" exitCode=0
Nov 03 19:51:44.225690 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.225554    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65d589b9db-prrwd" event={"ID":"4e4167c3-f632-477a-bc97-019193a3a692","Type":"ContainerDied","Data":"637895c1b3d99cedd7197a4cd971bfd91d1c63cb2ce0b8f260e4c1944504fa13"}
Nov 03 19:51:44.225690 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.225573    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65d589b9db-prrwd"
Nov 03 19:51:44.225690 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.225591    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65d589b9db-prrwd" event={"ID":"4e4167c3-f632-477a-bc97-019193a3a692","Type":"ContainerDied","Data":"21abad3a149991956b92bbe224fdf3b206efbb83746c77305dbdb734cfb5f3a2"}
Nov 03 19:51:44.236662 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.236641    5591 scope.go:122] "RemoveContainer" containerID="5453b0e62130643b6496895f5c5e0589c6930eb5f50000344ca55de9137c62de"
Nov 03 19:51:44.237056 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:51:44.236942    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5453b0e62130643b6496895f5c5e0589c6930eb5f50000344ca55de9137c62de\": container with ID starting with 5453b0e62130643b6496895f5c5e0589c6930eb5f50000344ca55de9137c62de not found: ID does not exist" containerID="5453b0e62130643b6496895f5c5e0589c6930eb5f50000344ca55de9137c62de"
Nov 03 19:51:44.237056 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.236974    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5453b0e62130643b6496895f5c5e0589c6930eb5f50000344ca55de9137c62de"} err="failed to get container status \"5453b0e62130643b6496895f5c5e0589c6930eb5f50000344ca55de9137c62de\": rpc error: code = NotFound desc = could not find container \"5453b0e62130643b6496895f5c5e0589c6930eb5f50000344ca55de9137c62de\": container with ID starting with 5453b0e62130643b6496895f5c5e0589c6930eb5f50000344ca55de9137c62de not found: ID does not exist"
Nov 03 19:51:44.237056 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.236994    5591 scope.go:122] "RemoveContainer" containerID="637895c1b3d99cedd7197a4cd971bfd91d1c63cb2ce0b8f260e4c1944504fa13"
Nov 03 19:51:44.247299 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.247272    5591 scope.go:122] "RemoveContainer" containerID="637895c1b3d99cedd7197a4cd971bfd91d1c63cb2ce0b8f260e4c1944504fa13"
Nov 03 19:51:44.247634 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:51:44.247605    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"637895c1b3d99cedd7197a4cd971bfd91d1c63cb2ce0b8f260e4c1944504fa13\": container with ID starting with 637895c1b3d99cedd7197a4cd971bfd91d1c63cb2ce0b8f260e4c1944504fa13 not found: ID does not exist" containerID="637895c1b3d99cedd7197a4cd971bfd91d1c63cb2ce0b8f260e4c1944504fa13"
Nov 03 19:51:44.247724 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.247637    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"637895c1b3d99cedd7197a4cd971bfd91d1c63cb2ce0b8f260e4c1944504fa13"} err="failed to get container status \"637895c1b3d99cedd7197a4cd971bfd91d1c63cb2ce0b8f260e4c1944504fa13\": rpc error: code = NotFound desc = could not find container \"637895c1b3d99cedd7197a4cd971bfd91d1c63cb2ce0b8f260e4c1944504fa13\": container with ID starting with 637895c1b3d99cedd7197a4cd971bfd91d1c63cb2ce0b8f260e4c1944504fa13 not found: ID does not exist"
Nov 03 19:51:44.250377 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.250356    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-79986d969f-ch5w8"]
Nov 03 19:51:44.254461 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.254438    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-79986d969f-ch5w8"]
Nov 03 19:51:44.266320 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.266294    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-65d589b9db-prrwd"]
Nov 03 19:51:44.268499 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.268481    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-65d589b9db-prrwd"]
Nov 03 19:51:44.285022 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:51:44.284990    5591 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6149f17_bdd7_44fb_a7dc_1314f89a38a6.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4167c3_f632_477a_bc97_019193a3a692.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4167c3_f632_477a_bc97_019193a3a692.slice/crio-21abad3a149991956b92bbe224fdf3b206efbb83746c77305dbdb734cfb5f3a2\": RecentStats: unable to find data in memory cache]"
Nov 03 19:51:44.332550 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.332514    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"]
Nov 03 19:51:44.333060 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.333047    5591 cpu_manager.go:414] "containerMap: removing container" podUID="4e4167c3-f632-477a-bc97-019193a3a692" containerName="payload-processing"
Nov 03 19:51:44.333114 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.333062    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e4167c3-f632-477a-bc97-019193a3a692" containerName="payload-processing"
Nov 03 19:51:44.333114 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.333075    5591 cpu_manager.go:414] "containerMap: removing container" podUID="a6149f17-bdd7-44fb-a7dc-1314f89a38a6" containerName="payload-processing"
Nov 03 19:51:44.333114 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.333080    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6149f17-bdd7-44fb-a7dc-1314f89a38a6" containerName="payload-processing"
Nov 03 19:51:44.333233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.333188    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e4167c3-f632-477a-bc97-019193a3a692" containerName="payload-processing"
Nov 03 19:51:44.333233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.333197    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6149f17-bdd7-44fb-a7dc-1314f89a38a6" containerName="payload-processing"
Nov 03 19:51:44.338183 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.338166    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.349228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.349193    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"]
Nov 03 19:51:44.513707 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.513673    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.513896 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.513752    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.513972 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.513915    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a74bc3-6a5f-46be-8094-74d859931e19-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.513972 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.513952    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a74bc3-6a5f-46be-8094-74d859931e19-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.514074 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.513979    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.514109 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.514090    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a74bc3-6a5f-46be-8094-74d859931e19-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.514255 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.514199    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.514410 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.514392    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.514448 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.514427    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4lfh\" (UniqueName: \"kubernetes.io/projected/73a74bc3-6a5f-46be-8094-74d859931e19-kube-api-access-k4lfh\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.615469 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.615439    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.615855 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.615510    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.615855 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.615606    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4lfh\" (UniqueName: \"kubernetes.io/projected/73a74bc3-6a5f-46be-8094-74d859931e19-kube-api-access-k4lfh\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.615855 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.615659    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.615855 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.615698    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.615855 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.615770    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a74bc3-6a5f-46be-8094-74d859931e19-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.615855 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.615794    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a74bc3-6a5f-46be-8094-74d859931e19-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.615855 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.615820    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.615855 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.615855    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a74bc3-6a5f-46be-8094-74d859931e19-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.616260 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.615880    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.616260 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.615930    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.616260 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.616030    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.616260 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.616166    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.616465 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.616448    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a74bc3-6a5f-46be-8094-74d859931e19-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.618359 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.618340    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.618826 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.618807    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a74bc3-6a5f-46be-8094-74d859931e19-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.627808 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.627784    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a74bc3-6a5f-46be-8094-74d859931e19-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.627935 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.627924    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4lfh\" (UniqueName: \"kubernetes.io/projected/73a74bc3-6a5f-46be-8094-74d859931e19-kube-api-access-k4lfh\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.654065 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.654040    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:44.940052 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.940027    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"]
Nov 03 19:51:44.941649 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:51:44.941625    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a74bc3_6a5f_46be_8094_74d859931e19.slice/crio-cf48637d75b977083841d6a7dc77235632d94f9bb2147633ba4492c234792c26 WatchSource:0}: Error finding container cf48637d75b977083841d6a7dc77235632d94f9bb2147633ba4492c234792c26: Status 404 returned error can't find the container with id cf48637d75b977083841d6a7dc77235632d94f9bb2147633ba4492c234792c26
Nov 03 19:51:44.943928 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.943898    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:51:44.943990 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.943960    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:51:44.944026 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:44.943989    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:51:45.235465 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:45.235373    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7" event={"ID":"73a74bc3-6a5f-46be-8094-74d859931e19","Type":"ContainerStarted","Data":"3d655a9905f2f1c41cdfb10e9f97ca1901907d541062e37be515ecb0ce36be60"}
Nov 03 19:51:45.235465 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:45.235435    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7" event={"ID":"73a74bc3-6a5f-46be-8094-74d859931e19","Type":"ContainerStarted","Data":"cf48637d75b977083841d6a7dc77235632d94f9bb2147633ba4492c234792c26"}
Nov 03 19:51:45.259072 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:45.259015    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7" podStartSLOduration=1.258998951 podStartE2EDuration="1.258998951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:45.257352287 +0000 UTC m=+854.029645308" watchObservedRunningTime="2026-11-03 19:51:45.258998951 +0000 UTC m=+854.031292021"
Nov 03 19:51:45.349433 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:45.349406    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e4167c3-f632-477a-bc97-019193a3a692" path="/var/lib/kubelet/pods/4e4167c3-f632-477a-bc97-019193a3a692/volumes"
Nov 03 19:51:45.349803 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:45.349791    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6149f17-bdd7-44fb-a7dc-1314f89a38a6" path="/var/lib/kubelet/pods/a6149f17-bdd7-44fb-a7dc-1314f89a38a6/volumes"
Nov 03 19:51:45.654879 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:45.654854    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:46.656612 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:46.656579    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 19:51:46.657031 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:46.656631    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7" podUID="73a74bc3-6a5f-46be-8094-74d859931e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 19:51:47.549983 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:47.549953    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:51:47.655308 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:47.655270    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 19:51:47.655510 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:47.655325    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7" podUID="73a74bc3-6a5f-46be-8094-74d859931e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 19:51:48.657589 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:48.657552    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 19:51:48.658256 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:48.658184    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7" podUID="73a74bc3-6a5f-46be-8094-74d859931e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 19:51:49.663345 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:49.663296    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:49.666440 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:49.666410    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 19:51:49.666653 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:49.666629    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7" podUID="73a74bc3-6a5f-46be-8094-74d859931e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 19:51:49.822592 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:49.822546    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:49.971525 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:49.971357    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:50.044358 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:50.044313    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 19:51:50.551339 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:50.551292    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:50.657231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:50.656624    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 19:51:50.657231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:50.656709    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7" podUID="73a74bc3-6a5f-46be-8094-74d859931e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 19:51:50.697238 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:50.664314    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:50.697238 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:50.664919    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:51:51.032353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:51.032303    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:51.033427 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:51.033405    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:51:51.526325 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:51.498165    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:51.612306 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:51.609512    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:51.670227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:51.654458    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:51:51.671480 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:51.666100    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 19:51:51.671691 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:51.671672    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7" podUID="73a74bc3-6a5f-46be-8094-74d859931e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 19:51:51.672084 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:51.667327    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:51.873643 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:51.873489    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:51:51.913920 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:51.913866    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:51:52.043228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:52.041284    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:52.150044 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:52.144273    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:52.688230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:52.675644    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 19:51:52.688230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:52.675706    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7" podUID="73a74bc3-6a5f-46be-8094-74d859931e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 19:51:52.688230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:52.676154    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:53.045394 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:53.045343    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:53.046778 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:53.046756    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:51:53.166231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:53.154594    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:53.166231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:53.155354    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:51:53.294230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:53.294167    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:51:53.587233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:53.562082    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:51:53.587674 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:53.587651    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:51:53.735004 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:53.734966    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 19:51:53.735319 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:53.735285    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7" podUID="73a74bc3-6a5f-46be-8094-74d859931e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 19:51:54.668236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:54.667750    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 19:51:54.668236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:54.667810    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7" podUID="73a74bc3-6a5f-46be-8094-74d859931e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 19:51:56.687709 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:56.687671    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:51:56.688267 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:56.687746    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7" podUID="73a74bc3-6a5f-46be-8094-74d859931e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:51:56.693916 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:56.693883    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:56.694332 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:56.694295    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:56.698279 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:56.698254    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 19:51:58.723764 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:51:58.723727    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:51:59.370531 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.370498    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"]
Nov 03 19:51:59.375860 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.375837    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.393525 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.393497    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"]
Nov 03 19:51:59.453793 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.453754    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.453972 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.453832    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.453972 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.453937    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.454049 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.453979    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.454049 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.454002    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wkjl\" (UniqueName: \"kubernetes.io/projected/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-kube-api-access-8wkjl\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.454112 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.454073    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.454190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.454149    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.454268 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.454192    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.454268 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.454260    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.555321 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.555284    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.555541 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.555338    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.555541 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.555376    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.555541 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.555396    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wkjl\" (UniqueName: \"kubernetes.io/projected/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-kube-api-access-8wkjl\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.555541 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.555430    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.555541 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.555475    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.555541 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.555535    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.555868 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.555575    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.555868 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.555635    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.556043 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.556016    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.556162 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.556137    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.556162 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.556157    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.556380 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.556142    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.556380 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.556335    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.558085 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.558068    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.558364 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.558345    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.567776 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.567749    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.568340 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.568309    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wkjl\" (UniqueName: \"kubernetes.io/projected/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-kube-api-access-8wkjl\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:51:59.714904 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:51:59.714823    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:52:00.040113 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:00.040081    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"]
Nov 03 19:52:00.042703 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:52:00.042671    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e0b0d0d_df65_4dfe_9333_1237cb41b8a4.slice/crio-e5b32a89e1a79e3794736fc4dbe5de7540209bae4bfdc000fe47765f86820c87 WatchSource:0}: Error finding container e5b32a89e1a79e3794736fc4dbe5de7540209bae4bfdc000fe47765f86820c87: Status 404 returned error can't find the container with id e5b32a89e1a79e3794736fc4dbe5de7540209bae4bfdc000fe47765f86820c87
Nov 03 19:52:00.045637 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:00.045606    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:52:00.045707 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:00.045672    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:52:00.045707 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:00.045701    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:52:00.865733 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:00.865697    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7" event={"ID":"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4","Type":"ContainerStarted","Data":"f451029784550ce6a6afc7134192e318ea92e5df08775922488f18a44b4363d0"}
Nov 03 19:52:00.865911 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:00.865756    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7" event={"ID":"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4","Type":"ContainerStarted","Data":"e5b32a89e1a79e3794736fc4dbe5de7540209bae4bfdc000fe47765f86820c87"}
Nov 03 19:52:01.715644 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:01.715610    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:52:01.717232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:01.717178    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 19:52:01.717353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:01.717271    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7" podUID="2e0b0d0d-df65-4dfe-9333-1237cb41b8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 19:52:02.351591 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:02.351529    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7" podStartSLOduration=3.351506797 podStartE2EDuration="3.351506797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:00.895099909 +0000 UTC m=+869.667392930" watchObservedRunningTime="2026-11-03 19:52:02.351506797 +0000 UTC m=+871.123799816"
Nov 03 19:52:02.352139 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:02.352115    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"]
Nov 03 19:52:02.357286 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:02.357266    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:02.363578 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:02.363550    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"distinct-2-w3-9f46e5-kserve-self-signed-certs\""
Nov 03 19:52:02.368161 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:02.368130    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"]
Nov 03 19:52:02.416162 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:02.416124    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d797881-05fe-4105-9dbe-e7ab7b768982-tls-certs\") pod \"distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc\" (UID: \"9d797881-05fe-4105-9dbe-e7ab7b768982\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:02.416382 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:02.416177    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t4zd\" (UniqueName: \"kubernetes.io/projected/9d797881-05fe-4105-9dbe-e7ab7b768982-kube-api-access-4t4zd\") pod \"distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc\" (UID: \"9d797881-05fe-4105-9dbe-e7ab7b768982\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:02.416382 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:02.416243    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-tmp-dir\") pod \"distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc\" (UID: \"9d797881-05fe-4105-9dbe-e7ab7b768982\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:02.416382 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:02.416291    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-dshm\") pod \"distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc\" (UID: \"9d797881-05fe-4105-9dbe-e7ab7b768982\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:02.416574 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:02.416409    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-home\") pod \"distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc\" (UID: \"9d797881-05fe-4105-9dbe-e7ab7b768982\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:02.416574 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:02.416519    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-model-cache\") pod \"distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc\" (UID: \"9d797881-05fe-4105-9dbe-e7ab7b768982\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:02.520181 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:02.520134    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-model-cache\") pod \"distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc\" (UID: \"9d797881-05fe-4105-9dbe-e7ab7b768982\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:02.520790 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:02.520750    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d797881-05fe-4105-9dbe-e7ab7b768982-tls-certs\") pod \"distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc\" (UID: \"9d797881-05fe-4105-9dbe-e7ab7b768982\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:02.521055 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:02.521022    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t4zd\" (UniqueName: \"kubernetes.io/projected/9d797881-05fe-4105-9dbe-e7ab7b768982-kube-api-access-4t4zd\") pod \"distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc\" (UID: \"9d797881-05fe-4105-9dbe-e7ab7b768982\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:02.529401 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:02.529347    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-tmp-dir\") pod \"distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc\" (UID: \"9d797881-05fe-4105-9dbe-e7ab7b768982\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:02.529709 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:02.529687    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-dshm\") pod \"distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc\" (UID: \"9d797881-05fe-4105-9dbe-e7ab7b768982\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:02.530017 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:02.529993    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-home\") pod \"distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc\" (UID: \"9d797881-05fe-4105-9dbe-e7ab7b768982\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:02.536915 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:02.530826    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-home\") pod \"distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc\" (UID: \"9d797881-05fe-4105-9dbe-e7ab7b768982\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:02.548757 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:02.537784    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-model-cache\") pod \"distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc\" (UID: \"9d797881-05fe-4105-9dbe-e7ab7b768982\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:02.558304 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:02.540586    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-tmp-dir\") pod \"distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc\" (UID: \"9d797881-05fe-4105-9dbe-e7ab7b768982\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:02.558535 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:02.545515    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-dshm\") pod \"distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc\" (UID: \"9d797881-05fe-4105-9dbe-e7ab7b768982\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:02.558629 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:02.552824    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d797881-05fe-4105-9dbe-e7ab7b768982-tls-certs\") pod \"distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc\" (UID: \"9d797881-05fe-4105-9dbe-e7ab7b768982\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:02.558729 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:02.552928    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t4zd\" (UniqueName: \"kubernetes.io/projected/9d797881-05fe-4105-9dbe-e7ab7b768982-kube-api-access-4t4zd\") pod \"distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc\" (UID: \"9d797881-05fe-4105-9dbe-e7ab7b768982\") " pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:02.680230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:02.680096    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:02.720082 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:02.719655    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 19:52:02.720082 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:02.719712    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7" podUID="2e0b0d0d-df65-4dfe-9333-1237cb41b8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 19:52:03.001975 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:03.001878    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"]
Nov 03 19:52:03.011285 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:52:03.011254    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d797881_05fe_4105_9dbe_e7ab7b768982.slice/crio-c51bb9976878005c8ebe27b1001d6bf38e3d648dd6589a498c1e957228427c72 WatchSource:0}: Error finding container c51bb9976878005c8ebe27b1001d6bf38e3d648dd6589a498c1e957228427c72: Status 404 returned error can't find the container with id c51bb9976878005c8ebe27b1001d6bf38e3d648dd6589a498c1e957228427c72
Nov 03 19:52:03.716001 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:03.715971    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 19:52:03.716162 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:03.716020    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7" podUID="2e0b0d0d-df65-4dfe-9333-1237cb41b8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 19:52:03.908942 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:03.908905    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" event={"ID":"9d797881-05fe-4105-9dbe-e7ab7b768982","Type":"ContainerStarted","Data":"0353ac9841442ed0b2b459195ab45ca668367f62a25e9870a28e1a8e48004e72"}
Nov 03 19:52:03.908942 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:03.908947    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" event={"ID":"9d797881-05fe-4105-9dbe-e7ab7b768982","Type":"ContainerStarted","Data":"c51bb9976878005c8ebe27b1001d6bf38e3d648dd6589a498c1e957228427c72"}
Nov 03 19:52:03.929616 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:03.929035    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podStartSLOduration=1.929019078 podStartE2EDuration="1.929019078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:03.928136994 +0000 UTC m=+872.700430013" watchObservedRunningTime="2026-11-03 19:52:03.929019078 +0000 UTC m=+872.701312098"
Nov 03 19:52:04.695226 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:04.684290    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:04.716258 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:04.716223    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 19:52:04.716440 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:04.716286    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7" podUID="2e0b0d0d-df65-4dfe-9333-1237cb41b8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 19:52:05.735393 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:05.735352    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 19:52:05.742241 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:05.735984    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7" podUID="2e0b0d0d-df65-4dfe-9333-1237cb41b8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 19:52:06.717707 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:06.717664    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 19:52:06.718011 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:06.717980    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7" podUID="2e0b0d0d-df65-4dfe-9333-1237cb41b8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 19:52:06.735232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:06.726324    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:06.876338 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:06.876293    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:52:07.317231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:07.311308    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:07.425232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:07.424304    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:07.727768 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:07.727666    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:07.751232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:07.728692    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:52:07.761239 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:07.752951    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 19:52:07.793134 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:07.792293    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7" podUID="2e0b0d0d-df65-4dfe-9333-1237cb41b8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 19:52:07.920232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:07.896480    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:07.920232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:07.897251    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:52:08.127330 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:08.109646    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": context deadline exceeded"
Nov 03 19:52:08.312228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:08.312165    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 19:52:08.330039 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:08.330009    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:52:08.490292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:08.483472    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:08.490292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:08.487370    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:52:08.719170 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:08.719133    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 19:52:08.741365 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:08.741269    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7" podUID="2e0b0d0d-df65-4dfe-9333-1237cb41b8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 19:52:08.741944 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:08.737579    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:08.901235 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:08.897381    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:08.989737 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:08.989678    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:09.211411 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:09.211363    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:09.339004 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:09.329889    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:09.432865 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:09.432830    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:52:09.433821 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:09.433748    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:52:09.734254 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:09.717568    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 19:52:09.734254 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:09.717630    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7" podUID="2e0b0d0d-df65-4dfe-9333-1237cb41b8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 19:52:09.742895 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:09.742839    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:09.824225 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:09.821413    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": context deadline exceeded"
Nov 03 19:52:10.119474 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:10.119430    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:52:10.355392 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:10.352030    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:10.508447 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:10.504968    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:52:11.143249 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:11.141316    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:52:11.143249 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:11.143036    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:52:11.144807 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:11.144740    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:52:12.681738 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:12.681696    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:12.694474 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:12.694449    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:13.543364 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:13.543313    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:16.683708 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.683671    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"]
Nov 03 19:52:16.699634 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.699605    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:16.736562 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.736528    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"]
Nov 03 19:52:16.765273 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.765241    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:16.765433 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.765289    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:16.765433 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.765352    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:16.765433 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.765375    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:16.765433 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.765400    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:16.765665 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.765437    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7tw5\" (UniqueName: \"kubernetes.io/projected/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-kube-api-access-b7tw5\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:16.765665 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.765513    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:16.765665 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.765552    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:16.765823 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.765710    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:16.867197 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.867156    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:16.867442 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.867424    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:16.867490 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.867461    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:16.867544 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.867530    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:16.867577 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.867560    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:16.867622 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.867577    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:16.867679 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.867615    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7tw5\" (UniqueName: \"kubernetes.io/projected/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-kube-api-access-b7tw5\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:16.867679 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.867664    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:16.867771 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.867698    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:16.867932 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.867911    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:16.868017 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.867984    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:16.868075 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.868020    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:16.868190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.868159    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:16.868484 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.868299    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:16.870138 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.870110    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:16.870280 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.870262    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:16.882630 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.882604    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:16.882790 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:16.882766    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7tw5\" (UniqueName: \"kubernetes.io/projected/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-kube-api-access-b7tw5\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:17.073985 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:17.073953    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:17.363465 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:17.363442    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"]
Nov 03 19:52:17.364979 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:52:17.364949    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod770df5ab_84a7_4fe9_999b_8ca9ee6464a4.slice/crio-19c02d52dd3fa71fb4ffd28939b23f5c55cb84988ad07176b125d66ff2b3be00 WatchSource:0}: Error finding container 19c02d52dd3fa71fb4ffd28939b23f5c55cb84988ad07176b125d66ff2b3be00: Status 404 returned error can't find the container with id 19c02d52dd3fa71fb4ffd28939b23f5c55cb84988ad07176b125d66ff2b3be00
Nov 03 19:52:17.367683 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:17.367647    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:52:17.367802 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:17.367724    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:52:17.367802 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:17.367767    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:52:17.569938 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:17.569896    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc" event={"ID":"770df5ab-84a7-4fe9-999b-8ca9ee6464a4","Type":"ContainerStarted","Data":"05986903284faf908de90f1f43aab54545c06a3e029af3a54a2d6c3731028307"}
Nov 03 19:52:17.570144 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:17.570004    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc" event={"ID":"770df5ab-84a7-4fe9-999b-8ca9ee6464a4","Type":"ContainerStarted","Data":"19c02d52dd3fa71fb4ffd28939b23f5c55cb84988ad07176b125d66ff2b3be00"}
Nov 03 19:52:17.596847 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:17.596789    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc" podStartSLOduration=1.596766373 podStartE2EDuration="1.596766373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:17.59301662 +0000 UTC m=+886.365309630" watchObservedRunningTime="2026-11-03 19:52:17.596766373 +0000 UTC m=+886.369059388"
Nov 03 19:52:18.074745 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:18.074709    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:18.076316 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:18.076285    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:52:18.076559 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:18.076532    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc" podUID="770df5ab-84a7-4fe9-999b-8ca9ee6464a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 19:52:18.632578 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:18.632543    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:52:19.074630 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:19.074599    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:52:19.074815 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:19.074650    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc" podUID="770df5ab-84a7-4fe9-999b-8ca9ee6464a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 19:52:20.074972 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:20.074654    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:52:20.074972 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:20.074716    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc" podUID="770df5ab-84a7-4fe9-999b-8ca9ee6464a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 19:52:21.076234 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:21.075933    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:52:21.076234 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:21.076015    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc" podUID="770df5ab-84a7-4fe9-999b-8ca9ee6464a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 19:52:21.511294 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:21.510372    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:21.747352 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:21.747296    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 19:52:21.909271 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:21.906404    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:52:21.921373 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:21.920500    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:22.087231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:22.086719    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:52:22.087231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:22.086789    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc" podUID="770df5ab-84a7-4fe9-999b-8ca9ee6464a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 19:52:22.357538 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:22.357472    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:22.462185 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:22.461578    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 19:52:22.534384 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:22.534325    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:22.758342 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:22.758256    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"]
Nov 03 19:52:22.807820 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:22.806159    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:22.840010 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:22.839980    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:52:22.844187 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:22.840187    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:52:22.907536 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:22.901409    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d3d0df\"/\"kube-root-ca.crt\""
Nov 03 19:52:22.907536 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:22.902329    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d3d0df\"/\"default-dockercfg-rgjbk\""
Nov 03 19:52:22.907536 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:22.902404    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d3d0df\"/\"distinct-2-w0-d3d0df-kserve-self-signed-certs\""
Nov 03 19:52:22.928252 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:22.921557    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:22.929971 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:22.929948    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:52:23.048641 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.048608    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-model-cache\") pod \"distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t\" (UID: \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:52:23.048977 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.048953    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-tmp-dir\") pod \"distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t\" (UID: \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:52:23.049142 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.049126    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-home\") pod \"distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t\" (UID: \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:52:23.049671 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.049645    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-dshm\") pod \"distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t\" (UID: \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:52:23.057231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.053482    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-tls-certs\") pod \"distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t\" (UID: \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:52:23.061226 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.057515    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c4cp\" (UniqueName: \"kubernetes.io/projected/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-kube-api-access-7c4cp\") pod \"distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t\" (UID: \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:52:23.077901 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.077865    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"]
Nov 03 19:52:23.171232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.163194    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-tls-certs\") pod \"distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t\" (UID: \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:52:23.273803 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.188364    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:52:23.302808 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.189049    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d3d0df\"/\"openshift-service-ca.crt\""
Nov 03 19:52:23.303104 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.277294    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c4cp\" (UniqueName: \"kubernetes.io/projected/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-kube-api-access-7c4cp\") pod \"distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t\" (UID: \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:52:23.304109 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.304080    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-model-cache\") pod \"distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t\" (UID: \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:52:23.308480 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.302301    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc" podUID="770df5ab-84a7-4fe9-999b-8ca9ee6464a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 19:52:23.315319 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.304720    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-model-cache\") pod \"distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t\" (UID: \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:52:23.315650 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.302013    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-tls-certs\") pod \"distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t\" (UID: \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:52:23.315793 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.313534    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-tmp-dir\") pod \"distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t\" (UID: \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:52:23.315910 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.315466    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-tmp-dir\") pod \"distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t\" (UID: \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:52:23.324523 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.324490    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-home\") pod \"distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t\" (UID: \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:52:23.345692 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.345663    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-dshm\") pod \"distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t\" (UID: \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:52:23.346486 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.346463    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-home\") pod \"distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t\" (UID: \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:52:23.365258 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.361755    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:23.403814 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.403739    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c4cp\" (UniqueName: \"kubernetes.io/projected/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-kube-api-access-7c4cp\") pod \"distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t\" (UID: \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:52:23.430903 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.427952    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-dshm\") pod \"distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t\" (UID: \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\") " pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:52:23.478650 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.472087    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:23.702292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.670647    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:52:23.702292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.693222    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 19:52:23.740328 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.740280    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:23.817229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.817169    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 19:52:23.957719 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:23.954732    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:24.030379 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:24.014775    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:52:24.030379 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:24.014842    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:52:24.030379 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:24.014873    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:24.030379 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:24.014901    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:24.095162 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:24.095116    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:52:24.095523 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:24.095481    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc" podUID="770df5ab-84a7-4fe9-999b-8ca9ee6464a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 19:52:24.364369 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:24.364320    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:24.473424 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:24.473373    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:24.630283 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:24.610007    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:24.914518 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:24.910710    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 19:52:25.003232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:24.956501    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 19:52:25.003232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:24.962274    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:52:25.095251 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:25.078128    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:52:25.095251 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:25.078238    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc" podUID="770df5ab-84a7-4fe9-999b-8ca9ee6464a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 19:52:25.174439 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:25.162560    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"]
Nov 03 19:52:25.282230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:25.280023    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:52:25.285227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:25.284995    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:52:25.287774 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:25.287244    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:52:25.507252 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:25.502290    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:52:26.055089 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:26.055051    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" event={"ID":"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862","Type":"ContainerStarted","Data":"d03a7ea2a9efea2faa83bd80d81895629158b902359440342412831d2a55aaaf"}
Nov 03 19:52:26.055272 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:26.055105    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" event={"ID":"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862","Type":"ContainerStarted","Data":"3bad33fc886df1cf3afc673c84900f3026f3c8bb8bcc001bd55cb67999db7731"}
Nov 03 19:52:26.074973 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:26.074935    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:52:26.075244 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:26.074997    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc" podUID="770df5ab-84a7-4fe9-999b-8ca9ee6464a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 19:52:26.075384 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:26.075239    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podStartSLOduration=4.075226831 podStartE2EDuration="4.075226831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:26.07505042 +0000 UTC m=+894.847343438" watchObservedRunningTime="2026-11-03 19:52:26.075226831 +0000 UTC m=+894.847519841"
Nov 03 19:52:26.708228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:26.704642    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:52:27.074872 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:27.074838    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:52:27.075060 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:27.074891    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc" podUID="770df5ab-84a7-4fe9-999b-8ca9ee6464a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 19:52:28.078782 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:28.078751    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:28.079166 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:28.079141    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:28.079965 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:28.079945    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:52:28.724505 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:52:28.724464    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:52:31.500870 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:31.500842    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 19:52:31.501344 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:31.501230    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 19:52:33.699632 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:33.699599    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:52:33.712510 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:33.712481    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:52:34.146728 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:34.146701    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:52:38.918232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:38.908796    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:38.963447 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:38.963364    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:39.142413 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:39.142352    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 19:52:39.167477 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:39.167422    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:39.305231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:39.304355    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:39.453364 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:39.453307    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:39.533781 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:39.533720    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:39.817230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:39.806320    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:39.915229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:39.914299    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:39.919228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:39.915752    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:52:39.932233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:39.928291    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:39.967344 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:39.967285    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 19:52:39.968510 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:39.968491    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:52:40.182805 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:40.177496    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:40.182805 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:40.180651    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:52:40.439290 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:40.370318    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 19:52:40.667232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:40.583770    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:40.667232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:40.585018    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 19:52:40.667232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:40.649380    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:52:40.667232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:40.650103    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:40.667232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:40.651231    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:52:41.065231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:41.030023    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:52:41.118229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:41.068647    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:41.174638 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:41.174571    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:41.347478 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:41.206421    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:41.359868 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:41.207253    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:52:41.692420 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:41.689869    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:41.692420 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:41.692404    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 19:52:41.692763 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:41.692722    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 19:52:41.832593 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:41.808562    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:42.214653 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:42.214610    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 19:52:42.404378 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:42.404327    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:42.438235 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:42.412935    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:42.438966 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:42.421746    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 19:52:42.697227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:42.690696    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:42.710238 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:42.705516    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:42.735419 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:42.735361    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:42.822441 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:42.822394    5591 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:52:42.822745 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:42.822697    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm" podUID="e31559fd-2ec2-48cd-8f00-8167a567097b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:43.248225 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:43.241700    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:43.422233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:43.415339    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:43.521283 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:43.519746    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 19:52:43.521283 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:43.520113    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 19:52:43.703232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:43.701279    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:43.753115 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:43.744118    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 19:52:43.763226 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:43.754591    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 19:52:43.763226 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:43.757554    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:52:43.837283 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:43.834954    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:44.342685 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.342623    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 19:52:44.455231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.452704    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:52:44.478874 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.478849    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"]
Nov 03 19:52:44.487462 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.487434    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.541802 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.538317    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 19:52:44.544285 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.544217    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:44.549251 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.545141    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"]
Nov 03 19:52:44.575864 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.575426    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:52:44.649851 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.649754    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded"
Nov 03 19:52:44.664740 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.664702    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.673291 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.673258    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01b87ad5-beef-41b7-99be-693434e57bbc-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.673477 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.673458    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.673750 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.673726    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b87ad5-beef-41b7-99be-693434e57bbc-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.673991 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.673968    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.674436 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.674412    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8spg\" (UniqueName: \"kubernetes.io/projected/01b87ad5-beef-41b7-99be-693434e57bbc-kube-api-access-z8spg\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.676668 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.675029    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.676668 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.675226    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.676668 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.675336    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b87ad5-beef-41b7-99be-693434e57bbc-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.679502 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.679481    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:52:44.722181 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.722145    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:44.758572 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.757853    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:44.759121 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.758856    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:44.776732 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.776693    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.776880 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.776796    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b87ad5-beef-41b7-99be-693434e57bbc-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.776880 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.776854    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.776995 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.776890    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01b87ad5-beef-41b7-99be-693434e57bbc-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.776995 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.776970    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.777109 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.777019    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b87ad5-beef-41b7-99be-693434e57bbc-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.777109 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.777056    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.777233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.777141    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8spg\" (UniqueName: \"kubernetes.io/projected/01b87ad5-beef-41b7-99be-693434e57bbc-kube-api-access-z8spg\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.777233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.777178    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.786112 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.786084    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.789844 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.786919    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b87ad5-beef-41b7-99be-693434e57bbc-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.789993 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.788046    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.790109 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.788282    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.790244 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.788480    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.792297 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.791727    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b87ad5-beef-41b7-99be-693434e57bbc-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.815565 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.815504    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01b87ad5-beef-41b7-99be-693434e57bbc-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.846618 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.845916    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"]
Nov 03 19:52:44.871228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.852933    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:44.873555 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.858624    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"]
Nov 03 19:52:44.882965 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.874108    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"]
Nov 03 19:52:44.882965 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.878049    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.882965 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.879391    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:44.882965 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.880456    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcvbg\" (UniqueName: \"kubernetes.io/projected/76f4135b-c979-49ee-a372-a68759b98276-kube-api-access-dcvbg\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:44.882965 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.880506    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:44.882965 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.880565    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:44.882965 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.880591    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f4135b-c979-49ee-a372-a68759b98276-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:44.882965 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.880655    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:44.882965 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.880725    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f4135b-c979-49ee-a372-a68759b98276-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:44.882965 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.880823    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f4135b-c979-49ee-a372-a68759b98276-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:44.882965 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.880859    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:44.889261 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.889171    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8spg\" (UniqueName: \"kubernetes.io/projected/01b87ad5-beef-41b7-99be-693434e57bbc-kube-api-access-z8spg\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:44.906572 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.905958    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:44.906572 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.906107    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"]
Nov 03 19:52:44.906572 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.906181    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:52:44.906845 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.906040    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:44.934354 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:44.934292    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:52:45.006233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.005988    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f4135b-c979-49ee-a372-a68759b98276-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:45.006233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.006046    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:45.006233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.006139    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:45.006233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.006167    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcvbg\" (UniqueName: \"kubernetes.io/projected/76f4135b-c979-49ee-a372-a68759b98276-kube-api-access-dcvbg\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:45.006233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.006237    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:45.006633 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.006331    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:45.006633 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.006356    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f4135b-c979-49ee-a372-a68759b98276-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:45.006633 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.006423    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:45.006633 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.006499    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f4135b-c979-49ee-a372-a68759b98276-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:45.027389 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.007498    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f4135b-c979-49ee-a372-a68759b98276-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:45.027389 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.007821    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:45.035580 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.035542    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:45.039225 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.036027    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:45.056384 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.036181    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:45.066743 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.066697    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:45.116096 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.079223    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:45.116574 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.094452    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcvbg\" (UniqueName: \"kubernetes.io/projected/76f4135b-c979-49ee-a372-a68759b98276-kube-api-access-dcvbg\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:45.124923 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.097320    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f4135b-c979-49ee-a372-a68759b98276-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:45.125166 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.099523    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f4135b-c979-49ee-a372-a68759b98276-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:45.126050 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.112097    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.126342 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.126282    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.128177 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.128153    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.128368 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.128346    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.128527 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.128508    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.128682 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.128664    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.128953 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.128923    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdflb\" (UniqueName: \"kubernetes.io/projected/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-kube-api-access-zdflb\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.129175 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.129155    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.129624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.129602    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.238651 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.237398    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.238651 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.237604    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.238651 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.237640    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.238651 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.237723    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.238651 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.237743    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.238651 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.237769    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.238651 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.237783    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.238651 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.237848    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdflb\" (UniqueName: \"kubernetes.io/projected/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-kube-api-access-zdflb\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.238651 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.237887    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.238651 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.238232    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.238651 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.238428    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:45.238651 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.238525    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.240290 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.239395    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.240290 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.239631    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.240290 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.240182    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.243103 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.241677    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.243748 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.243713    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.259480 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.259416    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdflb\" (UniqueName: \"kubernetes.io/projected/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-kube-api-access-zdflb\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.261740 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.261712    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.356733 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.356491    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:45.457246 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.456862    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:52:45.457246 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.456947    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:52:45.457246 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.456990    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:52:45.458546 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.458520    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"]
Nov 03 19:52:45.581700 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.581655    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx" event={"ID":"01b87ad5-beef-41b7-99be-693434e57bbc","Type":"ContainerStarted","Data":"7fa297fc2046c52f70dd6a0e4b6cc424e082d815df3ee0d96a1a40cd628d33bf"}
Nov 03 19:52:45.581876 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.581708    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx" event={"ID":"01b87ad5-beef-41b7-99be-693434e57bbc","Type":"ContainerStarted","Data":"70d7d40b78539ff429a361df47c78965f0a5ece0cda13068b1f7a86063b61749"}
Nov 03 19:52:45.607626 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.607593    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"]
Nov 03 19:52:45.612306 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:52:45.612249    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f4135b_c979_49ee_a372_a68759b98276.slice/crio-d2ec0abadaf7c8323afb904cf32ce3692bf439230cc24222aee7c4f3697ca536 WatchSource:0}: Error finding container d2ec0abadaf7c8323afb904cf32ce3692bf439230cc24222aee7c4f3697ca536: Status 404 returned error can't find the container with id d2ec0abadaf7c8323afb904cf32ce3692bf439230cc24222aee7c4f3697ca536
Nov 03 19:52:45.615962 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.615927    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:52:45.616083 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.616011    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:52:45.616083 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.616055    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:52:45.667150 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.667080    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx" podStartSLOduration=1.667061395 podStartE2EDuration="1.667061395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:45.661071893 +0000 UTC m=+914.433364913" watchObservedRunningTime="2026-11-03 19:52:45.667061395 +0000 UTC m=+914.439354414"
Nov 03 19:52:45.713840 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.713810    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"]
Nov 03 19:52:45.720637 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.720595    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:52:45.720734 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.720689    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:52:45.720786 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:45.720735    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:52:46.079468 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:46.079414    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:52:46.080804 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:46.080767    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 19:52:46.080949 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:46.080825    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx" podUID="01b87ad5-beef-41b7-99be-693434e57bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 19:52:46.666882 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:46.666844    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2" event={"ID":"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7","Type":"ContainerStarted","Data":"e047bc30be3d8c7dfdd94fdf0ed97f95a59fd061b9a20049a147f2b2e773476e"}
Nov 03 19:52:46.667368 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:46.666903    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2" event={"ID":"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7","Type":"ContainerStarted","Data":"b3105b478cf529d71f7ca943a1c3a66554f87a2da37cc5bead946961da445795"}
Nov 03 19:52:46.668393 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:46.668366    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs" event={"ID":"76f4135b-c979-49ee-a372-a68759b98276","Type":"ContainerStarted","Data":"24ea9acfd556d3c8ae1ef8317cadacc3fe210c949e594f097deb58509723f02d"}
Nov 03 19:52:46.668574 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:46.668416    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs" event={"ID":"76f4135b-c979-49ee-a372-a68759b98276","Type":"ContainerStarted","Data":"d2ec0abadaf7c8323afb904cf32ce3692bf439230cc24222aee7c4f3697ca536"}
Nov 03 19:52:46.732227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:46.720666    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2" podStartSLOduration=2.720643687 podStartE2EDuration="2.720643687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:46.700294553 +0000 UTC m=+915.472587574" watchObservedRunningTime="2026-11-03 19:52:46.720643687 +0000 UTC m=+915.492936706"
Nov 03 19:52:46.752231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:46.736814    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs" podStartSLOduration=2.736792488 podStartE2EDuration="2.736792488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:46.726871643 +0000 UTC m=+915.499164661" watchObservedRunningTime="2026-11-03 19:52:46.736792488 +0000 UTC m=+915.509085504"
Nov 03 19:52:47.095834 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:47.095797    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 19:52:47.096139 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:47.096106    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx" podUID="01b87ad5-beef-41b7-99be-693434e57bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 19:52:47.249374 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:47.244867    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:47.249374 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:47.245046    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:52:47.249374 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:47.245094    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs" podUID="76f4135b-c979-49ee-a372-a68759b98276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 19:52:47.358124 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:47.357987    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 19:52:47.358124 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:47.358047    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2" podUID="4b5fe641-913e-4f0d-81b6-1b6e2970e2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 19:52:47.365673 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:47.365649    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:48.100232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:48.095626    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 19:52:48.100232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:48.095695    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx" podUID="01b87ad5-beef-41b7-99be-693434e57bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 19:52:48.247280 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:48.243492    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:52:48.247280 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:48.243556    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs" podUID="76f4135b-c979-49ee-a372-a68759b98276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 19:52:48.359298 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:48.357624    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 19:52:48.359298 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:48.357689    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2" podUID="4b5fe641-913e-4f0d-81b6-1b6e2970e2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 19:52:49.095232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:49.082580    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 19:52:49.095232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:49.082645    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx" podUID="01b87ad5-beef-41b7-99be-693434e57bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 19:52:49.251231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:49.246435    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:52:49.251231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:49.246498    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs" podUID="76f4135b-c979-49ee-a372-a68759b98276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 19:52:49.357288 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:49.357168    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 19:52:49.357288 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:49.357248    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2" podUID="4b5fe641-913e-4f0d-81b6-1b6e2970e2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 19:52:50.099231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:50.091424    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 19:52:50.099231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:50.091485    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx" podUID="01b87ad5-beef-41b7-99be-693434e57bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 19:52:50.258625 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:50.258564    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:52:50.259014 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:50.258625    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs" podUID="76f4135b-c979-49ee-a372-a68759b98276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 19:52:50.362294 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:50.358352    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 19:52:50.362294 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:50.358411    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2" podUID="4b5fe641-913e-4f0d-81b6-1b6e2970e2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 19:52:51.086233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:51.080567    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 19:52:51.086233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:51.080629    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx" podUID="01b87ad5-beef-41b7-99be-693434e57bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 19:52:51.247452 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:51.247413    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:52:51.247765 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:51.247721    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs" podUID="76f4135b-c979-49ee-a372-a68759b98276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 19:52:51.365287 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:51.364307    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 19:52:51.365287 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:51.364365    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2" podUID="4b5fe641-913e-4f0d-81b6-1b6e2970e2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 19:52:51.969355 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:51.969295    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:52:52.141225 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:52.140624    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 19:52:52.141225 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:52.140685    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx" podUID="01b87ad5-beef-41b7-99be-693434e57bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 19:52:52.242682 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:52.242595    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:52:52.242921 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:52.242891    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs" podUID="76f4135b-c979-49ee-a372-a68759b98276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 19:52:52.357806 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:52.357769    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 19:52:52.358016 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:52.357832    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2" podUID="4b5fe641-913e-4f0d-81b6-1b6e2970e2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 19:52:53.142238 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:53.126710    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 19:52:53.167358 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:53.167313    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx" podUID="01b87ad5-beef-41b7-99be-693434e57bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 19:52:53.247877 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:53.247839    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:52:53.248136 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:53.248104    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs" podUID="76f4135b-c979-49ee-a372-a68759b98276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 19:52:53.356943 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:53.356908    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 19:52:53.362357 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:53.360946    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2" podUID="4b5fe641-913e-4f0d-81b6-1b6e2970e2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 19:52:54.080766 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:54.080711    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 19:52:54.081104 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:54.081059    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx" podUID="01b87ad5-beef-41b7-99be-693434e57bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 19:52:54.347286 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:54.343820    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:52:54.363357 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:54.363296    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs" podUID="76f4135b-c979-49ee-a372-a68759b98276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 19:52:54.363958 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:54.360098    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 19:52:54.364145 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:54.364125    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2" podUID="4b5fe641-913e-4f0d-81b6-1b6e2970e2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 19:52:54.445374 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:54.445295    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 19:52:54.556345 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:54.556285    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:54.614828 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:54.614673    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded"
Nov 03 19:52:54.697354 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:54.697307    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:54.729340 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:54.729298    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": context deadline exceeded"
Nov 03 19:52:54.775639 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:54.774277    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:55.369601 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:55.369539    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 19:52:55.371318 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:55.371276    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:52:55.464900 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:55.464853    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:55.466015 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:55.465991    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:52:55.611236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:55.564992    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:55.697327 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:55.697230    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:52:55.910584 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:55.910505    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 19:52:55.969315 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:55.912271    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:55.969315 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:55.951680    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:56.134400 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:56.134357    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:52:56.134695 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:56.134665    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx" podUID="01b87ad5-beef-41b7-99be-693434e57bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:56.396751 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:56.396707    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:56.432734 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:56.432700    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:52:56.433934 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:56.401347    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:52:56.434159 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:56.434134    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs" podUID="76f4135b-c979-49ee-a372-a68759b98276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:56.443449 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:56.402828    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 19:52:56.444166 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:56.444143    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:52:56.455957 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:56.455916    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:52:56.456266 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:56.456232    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2" podUID="4b5fe641-913e-4f0d-81b6-1b6e2970e2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:56.503660 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:56.503617    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:56.718101 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:56.718008    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:56.783313 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:56.781451    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:56.960439 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:56.960373    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 19:52:56.961584 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:56.961550    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:52:57.021408 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:57.021347    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:57.226232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:57.222494    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:52:57.231541 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:57.231451    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx" podUID="01b87ad5-beef-41b7-99be-693434e57bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:57.260612 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:57.257913    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:52:57.264229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:57.261002    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7" podUID="2e0b0d0d-df65-4dfe-9333-1237cb41b8a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:57.456960 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:57.456859    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:52:57.515003 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:57.514963    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:57.520059 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:57.519460    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:52:57.520059 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:57.519515    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs" podUID="76f4135b-c979-49ee-a372-a68759b98276" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:57.520620 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:57.520595    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:52:57.520795 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:57.520772    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2" podUID="4b5fe641-913e-4f0d-81b6-1b6e2970e2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:57.713309 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:57.707668    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 19:52:57.730075 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:57.719764    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:57.798780 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:57.787343    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 19:52:57.798780 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:57.797332    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:52:57.858540 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:57.858115    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:52:57.858540 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:57.858189    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7" podUID="73a74bc3-6a5f-46be-8094-74d859931e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:57.947400 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:57.944750    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:57.947400 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:57.946417    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:57.965155 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:57.965060    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:58.043554 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:57.981648    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 19:52:58.043862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:58.023312    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 19:52:58.148702 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:58.148651    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:58.295238 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:58.275025    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:58.295238 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:58.276405    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:58.314248 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:58.310615    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 19:52:58.340234 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:58.314702    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:52:58.386270 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:58.334328    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:52:58.411230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:58.386531    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx" podUID="01b87ad5-beef-41b7-99be-693434e57bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:58.467567 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:58.466444    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:58.523226 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:58.520302    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 19:52:58.716291 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:58.713217    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:58.725228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:58.721566    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 19:52:59.044377 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:59.044331    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:59.066232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:59.062743    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 19:52:59.184497 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:59.170980    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:59.196285 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:52:59.194381    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:52:59.319408 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:59.319305    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:59.470238 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:59.464386    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:52:59.483347 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:59.483301    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx" podUID="01b87ad5-beef-41b7-99be-693434e57bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:59.485032 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:59.485001    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:52:59.522468 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:59.522421    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 19:52:59.715399 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:59.715260    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:52:59.730636 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:59.721750    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:52:59.731008 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:59.730993    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:52:59.752158 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:59.752127    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:52:59.819354 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:52:59.819298    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:00.082230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:00.081892    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:53:00.298513 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:00.298482    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:53:00.336566 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:00.336475    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:00.337626 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:00.337602    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:53:00.522928 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:00.522884    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:00.523909 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:00.523884    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:53:00.621991 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:00.621903    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:53:00.622293 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:00.622262    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx" podUID="01b87ad5-beef-41b7-99be-693434e57bbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:00.635710 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:00.635685    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:53:00.635946 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:00.635935    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:53:00.636505 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:00.636488    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:53:01.155614 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:01.155583    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:53:01.787678 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:01.787644    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"]
Nov 03 19:53:01.793955 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:01.793929    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:01.804429 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:01.804396    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"]
Nov 03 19:53:01.914256 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:01.914198    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:01.914466 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:01.914326    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83260a88-c456-4334-81ec-f640ab303cc3-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:01.914466 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:01.914396    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjpr8\" (UniqueName: \"kubernetes.io/projected/83260a88-c456-4334-81ec-f640ab303cc3-kube-api-access-jjpr8\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:01.914466 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:01.914457    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:01.914656 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:01.914481    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83260a88-c456-4334-81ec-f640ab303cc3-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:01.914656 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:01.914602    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:01.914793 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:01.914774    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:01.914861 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:01.914813    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83260a88-c456-4334-81ec-f640ab303cc3-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:01.914920 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:01.914880    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:01.995801 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:01.995768    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj"]
Nov 03 19:53:02.000181 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.000158    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj"
Nov 03 19:53:02.013266 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.013134    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d8d238bf\""
Nov 03 19:53:02.014377 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.014354    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj"]
Nov 03 19:53:02.016181 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.016154    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:02.016339 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.016284    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:02.016339 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.016330    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83260a88-c456-4334-81ec-f640ab303cc3-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:02.016460 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.016370    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjpr8\" (UniqueName: \"kubernetes.io/projected/83260a88-c456-4334-81ec-f640ab303cc3-kube-api-access-jjpr8\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:02.016460 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.016444    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:02.016536 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.016472    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83260a88-c456-4334-81ec-f640ab303cc3-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:02.016536 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.016510    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:02.016712 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.016694    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:02.016836 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.016746    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83260a88-c456-4334-81ec-f640ab303cc3-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:02.017697 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.017673    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:02.018088 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.018068    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:02.041850 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.041752    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:02.042261 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.042228    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83260a88-c456-4334-81ec-f640ab303cc3-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:02.043620 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.043597    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:02.051864 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.051789    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g"]
Nov 03 19:53:02.062806 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.062741    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjpr8\" (UniqueName: \"kubernetes.io/projected/83260a88-c456-4334-81ec-f640ab303cc3-kube-api-access-jjpr8\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:02.063666 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.063611    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:02.064381 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.064360    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83260a88-c456-4334-81ec-f640ab303cc3-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:02.071538 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.071515    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g"]
Nov 03 19:53:02.071706 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.071638    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g"
Nov 03 19:53:02.075125 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.074991    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83260a88-c456-4334-81ec-f640ab303cc3-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:02.076265 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.076245    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d8d238bf-dockercfg-2k6xr\""
Nov 03 19:53:02.118333 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.118292    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a7f19b-214a-4785-b023-9311cffdc24f-plugins-config-volume\") pod \"payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj\" (UID: \"b5a7f19b-214a-4785-b023-9311cffdc24f\") " pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj"
Nov 03 19:53:02.118503 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.118373    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8kp2\" (UniqueName: \"kubernetes.io/projected/268bfdfd-019e-4a89-b66f-921a65dfa5ec-kube-api-access-c8kp2\") pod \"payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g\" (UID: \"268bfdfd-019e-4a89-b66f-921a65dfa5ec\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g"
Nov 03 19:53:02.118663 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.118635    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268bfdfd-019e-4a89-b66f-921a65dfa5ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g\" (UID: \"268bfdfd-019e-4a89-b66f-921a65dfa5ec\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g"
Nov 03 19:53:02.118805 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.118709    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nnfb\" (UniqueName: \"kubernetes.io/projected/b5a7f19b-214a-4785-b023-9311cffdc24f-kube-api-access-8nnfb\") pod \"payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj\" (UID: \"b5a7f19b-214a-4785-b023-9311cffdc24f\") " pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj"
Nov 03 19:53:02.134588 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.134557    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:02.280413 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.237564    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8kp2\" (UniqueName: \"kubernetes.io/projected/268bfdfd-019e-4a89-b66f-921a65dfa5ec-kube-api-access-c8kp2\") pod \"payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g\" (UID: \"268bfdfd-019e-4a89-b66f-921a65dfa5ec\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g"
Nov 03 19:53:02.280413 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.237850    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268bfdfd-019e-4a89-b66f-921a65dfa5ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g\" (UID: \"268bfdfd-019e-4a89-b66f-921a65dfa5ec\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g"
Nov 03 19:53:02.280413 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.237947    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nnfb\" (UniqueName: \"kubernetes.io/projected/b5a7f19b-214a-4785-b023-9311cffdc24f-kube-api-access-8nnfb\") pod \"payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj\" (UID: \"b5a7f19b-214a-4785-b023-9311cffdc24f\") " pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj"
Nov 03 19:53:02.280413 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.238053    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a7f19b-214a-4785-b023-9311cffdc24f-plugins-config-volume\") pod \"payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj\" (UID: \"b5a7f19b-214a-4785-b023-9311cffdc24f\") " pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj"
Nov 03 19:53:02.384237 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.324889    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268bfdfd-019e-4a89-b66f-921a65dfa5ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g\" (UID: \"268bfdfd-019e-4a89-b66f-921a65dfa5ec\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g"
Nov 03 19:53:02.384237 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.328822    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a7f19b-214a-4785-b023-9311cffdc24f-plugins-config-volume\") pod \"payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj\" (UID: \"b5a7f19b-214a-4785-b023-9311cffdc24f\") " pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj"
Nov 03 19:53:02.410227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.408911    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nnfb\" (UniqueName: \"kubernetes.io/projected/b5a7f19b-214a-4785-b023-9311cffdc24f-kube-api-access-8nnfb\") pod \"payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj\" (UID: \"b5a7f19b-214a-4785-b023-9311cffdc24f\") " pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj"
Nov 03 19:53:02.413428 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.413362    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8kp2\" (UniqueName: \"kubernetes.io/projected/268bfdfd-019e-4a89-b66f-921a65dfa5ec-kube-api-access-c8kp2\") pod \"payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g\" (UID: \"268bfdfd-019e-4a89-b66f-921a65dfa5ec\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g"
Nov 03 19:53:02.636095 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.636029    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"]
Nov 03 19:53:02.637128 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:53:02.637101    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83260a88_c456_4334_81ec_f640ab303cc3.slice/crio-395a6eef60cce290675629ab0af9f7c9e4dff70ff2dbbaff1b93224e02322fb8 WatchSource:0}: Error finding container 395a6eef60cce290675629ab0af9f7c9e4dff70ff2dbbaff1b93224e02322fb8: Status 404 returned error can't find the container with id 395a6eef60cce290675629ab0af9f7c9e4dff70ff2dbbaff1b93224e02322fb8
Nov 03 19:53:02.639830 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.639793    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:53:02.639937 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.639878    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:53:02.639937 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.639925    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:53:02.663521 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.663501    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj"
Nov 03 19:53:02.710516 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.710475    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g"
Nov 03 19:53:02.955704 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:02.955678    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj"]
Nov 03 19:53:02.957851 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:53:02.957824    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a7f19b_214a_4785_b023_9311cffdc24f.slice/crio-5aa2f1e8a7b89c8aad9f46bef0ce994d63a557994b18dc8210da34730e07b2d9 WatchSource:0}: Error finding container 5aa2f1e8a7b89c8aad9f46bef0ce994d63a557994b18dc8210da34730e07b2d9: Status 404 returned error can't find the container with id 5aa2f1e8a7b89c8aad9f46bef0ce994d63a557994b18dc8210da34730e07b2d9
Nov 03 19:53:03.054385 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:53:03.054350    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod268bfdfd_019e_4a89_b66f_921a65dfa5ec.slice/crio-d52c1ef97b6e5056644f16662503227a83041edb37b5735af5403076d877f642 WatchSource:0}: Error finding container d52c1ef97b6e5056644f16662503227a83041edb37b5735af5403076d877f642: Status 404 returned error can't find the container with id d52c1ef97b6e5056644f16662503227a83041edb37b5735af5403076d877f642
Nov 03 19:53:03.054533 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:03.054512    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g"]
Nov 03 19:53:03.263734 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:03.263285    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj" event={"ID":"b5a7f19b-214a-4785-b023-9311cffdc24f","Type":"ContainerStarted","Data":"20a19c5b76405e3d40192a31e3322231d293171d9b50907b5867c01c33f67b88"}
Nov 03 19:53:03.263734 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:03.263349    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj" event={"ID":"b5a7f19b-214a-4785-b023-9311cffdc24f","Type":"ContainerStarted","Data":"5aa2f1e8a7b89c8aad9f46bef0ce994d63a557994b18dc8210da34730e07b2d9"}
Nov 03 19:53:03.264445 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:03.264382    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj"
Nov 03 19:53:03.267507 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:03.267477    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g" event={"ID":"268bfdfd-019e-4a89-b66f-921a65dfa5ec","Type":"ContainerStarted","Data":"d52c1ef97b6e5056644f16662503227a83041edb37b5735af5403076d877f642"}
Nov 03 19:53:03.271333 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:03.270435    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp" event={"ID":"83260a88-c456-4334-81ec-f640ab303cc3","Type":"ContainerStarted","Data":"63b12d248fe3fede28096729d2efe8300f62ebe8f60c2125caf04ad205228c89"}
Nov 03 19:53:03.271333 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:03.270488    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp" event={"ID":"83260a88-c456-4334-81ec-f640ab303cc3","Type":"ContainerStarted","Data":"395a6eef60cce290675629ab0af9f7c9e4dff70ff2dbbaff1b93224e02322fb8"}
Nov 03 19:53:03.305558 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:03.305484    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj" podStartSLOduration=2.305459707 podStartE2EDuration="2.305459707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:03.28362572 +0000 UTC m=+932.055918738" watchObservedRunningTime="2026-11-03 19:53:03.305459707 +0000 UTC m=+932.077752725"
Nov 03 19:53:03.330358 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:03.330307    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp" podStartSLOduration=2.3302905 podStartE2EDuration="2.3302905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:03.328808109 +0000 UTC m=+932.101101130" watchObservedRunningTime="2026-11-03 19:53:03.3302905 +0000 UTC m=+932.102583520"
Nov 03 19:53:04.144155 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:04.142619    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:04.149950 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:04.149563    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 19:53:04.149950 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:04.149628    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp" podUID="83260a88-c456-4334-81ec-f640ab303cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 19:53:04.303672 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:04.303636    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g" event={"ID":"268bfdfd-019e-4a89-b66f-921a65dfa5ec","Type":"ContainerStarted","Data":"f9b86501d79c9b9ca9571dc9c07f7d121ab5b567a0dff865bfb1e071225b2c73"}
Nov 03 19:53:04.325146 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:04.325097    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g" podStartSLOduration=2.325083435 podStartE2EDuration="2.325083435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:04.322129124 +0000 UTC m=+933.094422142" watchObservedRunningTime="2026-11-03 19:53:04.325083435 +0000 UTC m=+933.097376519"
Nov 03 19:53:05.135136 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:05.135094    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 19:53:05.135349 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:05.135171    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp" podUID="83260a88-c456-4334-81ec-f640ab303cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 19:53:05.311241 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:05.311193    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g"
Nov 03 19:53:06.135842 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:06.135806    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 19:53:06.137910 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:06.135892    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp" podUID="83260a88-c456-4334-81ec-f640ab303cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 19:53:07.140233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:07.138680    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 19:53:07.140233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:07.138741    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp" podUID="83260a88-c456-4334-81ec-f640ab303cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 19:53:08.135425 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:08.135387    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 19:53:08.135617 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:08.135462    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp" podUID="83260a88-c456-4334-81ec-f640ab303cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 19:53:09.135388 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:09.135350    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 19:53:09.135802 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:09.135419    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp" podUID="83260a88-c456-4334-81ec-f640ab303cc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 19:53:10.453700 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:10.453656    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:10.454178 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:10.454093    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:10.454967 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:10.454945    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 19:53:14.342231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:14.341358    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:53:14.342231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:14.341450    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989" podUID="1faa21e4-5bf7-418d-adc8-3fc5bf955010" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:14.494231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:14.490365    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:53:14.532425 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:14.532377    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 19:53:14.704294 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:14.703313    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded"
Nov 03 19:53:14.707348 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:14.707302    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 19:53:14.718352 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:14.718300    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:14.744366 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:14.744307    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:14.771227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:14.766308    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:15.039232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:15.024405    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:15.554290 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:15.534411    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:53:15.681375 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:15.681318    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:15.738233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:15.725441    5591 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:53:15.738233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:15.725518    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr" podUID="b043886c-68e1-48d8-9ec4-8ecd4601a753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:15.738233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:15.725850    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:15.738233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:15.725924    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:15.762590 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:15.762471    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:15.837337 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:15.817539    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj"
Nov 03 19:53:15.899010 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:15.871133    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:53:15.899010 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:15.872325    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:53:15.973755 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:15.902324    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:53:15.973755 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:15.902407    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:53:16.093185 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:16.064895    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:16.397290 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:16.367356    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:53:16.417713 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:16.417683    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g"
Nov 03 19:53:16.551242 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:16.551170    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:16.829239 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:16.826293    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 19:53:16.829239 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:16.827265    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 19:53:16.881240 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:16.879383    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:16.881240 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:16.879907    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:16.912231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:16.908292    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 19:53:16.912231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:16.911094    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:53:17.089283 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:17.087299    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"]
Nov 03 19:53:17.132665 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:17.132636    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:17.419406 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:17.395721    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"]
Nov 03 19:53:17.419406 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:17.395893    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:17.581232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:17.555962    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:17.895279 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:17.880891    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 19:53:17.895279 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:17.881650    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 19:53:17.900386 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:17.900346    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:17.941852 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:17.931068    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 19:53:18.118311 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:17.963159    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.118311 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.116980    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.118311 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.117064    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3377787a-22d0-45ce-9c24-c39c840af373-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.118311 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.117180    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.118311 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.117420    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.118311 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.117574    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3377787a-22d0-45ce-9c24-c39c840af373-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.118311 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.117636    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3377787a-22d0-45ce-9c24-c39c840af373-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.118311 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.117717    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.118311 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.117788    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-766zv\" (UniqueName: \"kubernetes.io/projected/3377787a-22d0-45ce-9c24-c39c840af373-kube-api-access-766zv\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.256231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.250589    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": context deadline exceeded"
Nov 03 19:53:18.350231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.349756    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.350231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.349917    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3377787a-22d0-45ce-9c24-c39c840af373-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.350231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.349958    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3377787a-22d0-45ce-9c24-c39c840af373-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.350231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.350002    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.350231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.350058    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-766zv\" (UniqueName: \"kubernetes.io/projected/3377787a-22d0-45ce-9c24-c39c840af373-kube-api-access-766zv\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.350231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.350165    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.350231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.350242    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.350795 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.350303    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3377787a-22d0-45ce-9c24-c39c840af373-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.350795 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.350400    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.376287 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.351051    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.376287 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.356681    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3377787a-22d0-45ce-9c24-c39c840af373-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.376287 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.358657    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.376287 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.359079    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.376689 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.376602    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.441369 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.438987    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 19:53:18.529759 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.529724    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3377787a-22d0-45ce-9c24-c39c840af373-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.566887 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.558359    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:53:18.566887 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.559733    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3377787a-22d0-45ce-9c24-c39c840af373-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.649229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.648638    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.681244 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.678287    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-766zv\" (UniqueName: \"kubernetes.io/projected/3377787a-22d0-45ce-9c24-c39c840af373-kube-api-access-766zv\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.697287 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:53:18.683236    5591 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.091s"
Nov 03 19:53:18.767231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.737858    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:18.869397 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.869360    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 19:53:18.889799 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.881081    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:18.889799 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.882108    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:18.965723 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:18.956334    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:19.200732 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:19.200673    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 19:53:19.335294 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:19.289473    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:19.340234 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:19.335579    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-9cvzn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:53:19.340234 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:19.335644    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn" podUID="3a0f28bc-d05f-45bb-8431-3aaf73529510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:19.501240 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:19.471715    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 19:53:19.572690 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:19.572646    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:19.834393 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:19.829643    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:19.895282 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:19.881671    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:19.895282 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:19.882756    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 19:53:19.895282 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:19.883061    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 19:53:19.939657 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:19.939615    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:53:19.976787 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:19.976740    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:20.244275 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:20.237440    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:20.244275 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:20.242793    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 19:53:20.254502 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:20.250011    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:53:20.362303 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:53:20.362269    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3377787a_22d0_45ce_9c24_c39c840af373.slice/crio-5ad5c5030029d1a366dce8ba417c0b55ee2bfdbeadc97ebe3be8b6fb93ad51db WatchSource:0}: Error finding container 5ad5c5030029d1a366dce8ba417c0b55ee2bfdbeadc97ebe3be8b6fb93ad51db: Status 404 returned error can't find the container with id 5ad5c5030029d1a366dce8ba417c0b55ee2bfdbeadc97ebe3be8b6fb93ad51db
Nov 03 19:53:20.371229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:20.370467    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:53:20.402101 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:20.402076    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 19:53:20.402455 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:20.402438    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 19:53:20.424165 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:20.424128    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"]
Nov 03 19:53:20.430626 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:20.430592    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:53:20.444342 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:20.444312    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:53:20.455327 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:20.455282    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:53:20.455484 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:20.455377    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:53:20.455484 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:20.455425    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:53:20.499798 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:20.499717    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds" event={"ID":"3377787a-22d0-45ce-9c24-c39c840af373","Type":"ContainerStarted","Data":"5ad5c5030029d1a366dce8ba417c0b55ee2bfdbeadc97ebe3be8b6fb93ad51db"}
Nov 03 19:53:21.506981 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:21.506935    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds" event={"ID":"3377787a-22d0-45ce-9c24-c39c840af373","Type":"ContainerStarted","Data":"d07698bd0f99375147174643e3f8fd9636b15d9f5f0e389a08272a36c44231eb"}
Nov 03 19:53:21.535911 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:21.535848    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds" podStartSLOduration=5.535834274 podStartE2EDuration="5.535834274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:21.533718239 +0000 UTC m=+950.306011262" watchObservedRunningTime="2026-11-03 19:53:21.535834274 +0000 UTC m=+950.308127292"
Nov 03 19:53:21.741158 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:21.741114    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:21.742848 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:21.742816    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 19:53:21.742987 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:21.742882    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds" podUID="3377787a-22d0-45ce-9c24-c39c840af373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 19:53:22.741379 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:22.741343    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 19:53:22.741887 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:22.741406    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds" podUID="3377787a-22d0-45ce-9c24-c39c840af373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 19:53:23.776237 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:23.773798    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 19:53:23.776237 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:23.773883    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds" podUID="3377787a-22d0-45ce-9c24-c39c840af373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 19:53:24.741623 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:24.741584    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 19:53:24.741834 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:24.741663    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds" podUID="3377787a-22d0-45ce-9c24-c39c840af373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 19:53:25.743853 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:25.743814    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 19:53:25.744524 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:25.744493    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds" podUID="3377787a-22d0-45ce-9c24-c39c840af373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 19:53:26.740966 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:26.740929    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 19:53:26.741154 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:26.740991    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds" podUID="3377787a-22d0-45ce-9c24-c39c840af373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 19:53:27.224534 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:27.224497    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:53:27.741191 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:27.741152    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 19:53:27.741476 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:27.741257    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds" podUID="3377787a-22d0-45ce-9c24-c39c840af373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 19:53:28.759226 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:28.753655    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 03 19:53:28.759226 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:28.753726    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds" podUID="3377787a-22d0-45ce-9c24-c39c840af373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 03 19:53:28.782408 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:53:28.782364    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:53:29.469346 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:29.469298    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:29.813231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:29.812323    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:29.909230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:29.905311    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 19:53:30.417227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:30.416315    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:30.755158 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:30.755064    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:53:30.755462 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:30.755419    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds" podUID="3377787a-22d0-45ce-9c24-c39c840af373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:31.591398 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:31.590451    5591 scope.go:122] "RemoveContainer" containerID="2e2ff41d29f6d5f6ee79a17699b33af1b65c6b7ac909888eb0b18e1844f0c23a"
Nov 03 19:53:31.801227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:31.797801    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:53:31.801227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:31.797876    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds" podUID="3377787a-22d0-45ce-9c24-c39c840af373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:32.104911 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:32.104873    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:32.107304 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:32.106360    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:32.108962 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:32.107761    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:53:43.353819 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.353789    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"]
Nov 03 19:53:43.361603 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.361573    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.362895 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.362869    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"]
Nov 03 19:53:43.366080 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.366058    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-cc7720-openshift-default-dockercfg-slkkk\""
Nov 03 19:53:43.528236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.526386    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-workload-certs\") pod \"e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") " pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.528236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.526462    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-credential-socket\") pod \"e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") " pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.528236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.526537    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-data\") pod \"e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") " pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.528236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.526657    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-token\") pod \"e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") " pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.528236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.526699    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tsv4\" (UniqueName: \"kubernetes.io/projected/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-kube-api-access-8tsv4\") pod \"e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") " pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.528236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.526746    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istiod-ca-cert\") pod \"e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") " pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.528236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.526852    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-envoy\") pod \"e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") " pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.528236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.526903    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-workload-socket\") pod \"e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") " pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.528236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.526940    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-podinfo\") pod \"e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") " pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.639279 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.635457    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-data\") pod \"e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") " pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.639279 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.635612    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-token\") pod \"e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") " pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.639279 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.635665    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tsv4\" (UniqueName: \"kubernetes.io/projected/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-kube-api-access-8tsv4\") pod \"e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") " pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.639279 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.635726    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istiod-ca-cert\") pod \"e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") " pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.639279 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.635851    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-envoy\") pod \"e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") " pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.639279 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.635900    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-workload-socket\") pod \"e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") " pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.639279 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.635952    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-podinfo\") pod \"e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") " pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.639279 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.636050    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-workload-certs\") pod \"e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") " pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.639279 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.636116    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-credential-socket\") pod \"e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") " pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.639279 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.636737    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-credential-socket\") pod \"e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") " pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.639279 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.637108    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-data\") pod \"e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") " pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.656561 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.656510    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istiod-ca-cert\") pod \"e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") " pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.659573 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.659518    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-workload-certs\") pod \"e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") " pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.659856 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.659836    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-workload-socket\") pod \"e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") " pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.659995 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.659877    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-podinfo\") pod \"e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") " pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.660872 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.660520    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-envoy\") pod \"e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") " pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.660872 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.660828    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-token\") pod \"e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") " pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.681237 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.674921    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tsv4\" (UniqueName: \"kubernetes.io/projected/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-kube-api-access-8tsv4\") pod \"e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") " pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:43.703350 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:43.682766    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:44.115330 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:44.115301    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"]
Nov 03 19:53:44.116100 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:53:44.116069    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0e2592_7378_4f51_99b4_6b4f06ecc61c.slice/crio-369882f8d90f9a1411b883bebdd4a9c71cc37dbded8d7cd80be1b3269daf4b27 WatchSource:0}: Error finding container 369882f8d90f9a1411b883bebdd4a9c71cc37dbded8d7cd80be1b3269daf4b27: Status 404 returned error can't find the container with id 369882f8d90f9a1411b883bebdd4a9c71cc37dbded8d7cd80be1b3269daf4b27
Nov 03 19:53:44.118902 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:44.118859    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:53:44.119012 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:44.118943    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:53:44.119012 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:44.118992    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:53:44.866285 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:44.866246    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn" event={"ID":"fc0e2592-7378-4f51-99b4-6b4f06ecc61c","Type":"ContainerStarted","Data":"44754f7322be589979ca35b6e1215c8814e1921d6996837dc6c14aead87b490c"}
Nov 03 19:53:44.866700 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:44.866309    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn" event={"ID":"fc0e2592-7378-4f51-99b4-6b4f06ecc61c","Type":"ContainerStarted","Data":"369882f8d90f9a1411b883bebdd4a9c71cc37dbded8d7cd80be1b3269daf4b27"}
Nov 03 19:53:44.898372 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:44.898310    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn" podStartSLOduration=1.898290276 podStartE2EDuration="1.898290276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:44.895360371 +0000 UTC m=+973.667653390" watchObservedRunningTime="2026-11-03 19:53:44.898290276 +0000 UTC m=+973.670583293"
Nov 03 19:53:45.683457 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:45.683426    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:45.688971 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:45.688940    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:45.924142 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:45.924041    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:45.951346 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:45.945673    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:53:48.543944 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:48.543906    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz"]
Nov 03 19:53:48.548272 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:48.548248    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz"
Nov 03 19:53:48.556527 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:48.556504    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-cc7720\""
Nov 03 19:53:48.556954 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:48.556925    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-cc7720-dockercfg-6b4zk\""
Nov 03 19:53:48.563221 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:48.563185    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz"]
Nov 03 19:53:48.609718 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:48.609683    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97"]
Nov 03 19:53:48.614268 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:48.614243    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97"
Nov 03 19:53:48.614473 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:48.614445    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97"]
Nov 03 19:53:48.727238 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:48.712662    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727282fc-9c33-4102-96c1-de9b84f0d5d9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97\" (UID: \"727282fc-9c33-4102-96c1-de9b84f0d5d9\") " pod="openshift-ingress/payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97"
Nov 03 19:53:48.727577 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:48.727544    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7510db95-54ae-4fff-9a15-d54f80e27119-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz\" (UID: \"7510db95-54ae-4fff-9a15-d54f80e27119\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz"
Nov 03 19:53:48.727819 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:48.727802    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7j7t\" (UniqueName: \"kubernetes.io/projected/7510db95-54ae-4fff-9a15-d54f80e27119-kube-api-access-c7j7t\") pod \"payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz\" (UID: \"7510db95-54ae-4fff-9a15-d54f80e27119\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz"
Nov 03 19:53:48.735392 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:48.735353    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2jxf\" (UniqueName: \"kubernetes.io/projected/727282fc-9c33-4102-96c1-de9b84f0d5d9-kube-api-access-j2jxf\") pod \"payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97\" (UID: \"727282fc-9c33-4102-96c1-de9b84f0d5d9\") " pod="openshift-ingress/payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97"
Nov 03 19:53:48.838282 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:48.838091    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2jxf\" (UniqueName: \"kubernetes.io/projected/727282fc-9c33-4102-96c1-de9b84f0d5d9-kube-api-access-j2jxf\") pod \"payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97\" (UID: \"727282fc-9c33-4102-96c1-de9b84f0d5d9\") " pod="openshift-ingress/payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97"
Nov 03 19:53:48.838484 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:48.838404    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727282fc-9c33-4102-96c1-de9b84f0d5d9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97\" (UID: \"727282fc-9c33-4102-96c1-de9b84f0d5d9\") " pod="openshift-ingress/payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97"
Nov 03 19:53:48.838484 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:48.838446    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7510db95-54ae-4fff-9a15-d54f80e27119-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz\" (UID: \"7510db95-54ae-4fff-9a15-d54f80e27119\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz"
Nov 03 19:53:48.838616 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:48.838563    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7j7t\" (UniqueName: \"kubernetes.io/projected/7510db95-54ae-4fff-9a15-d54f80e27119-kube-api-access-c7j7t\") pod \"payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz\" (UID: \"7510db95-54ae-4fff-9a15-d54f80e27119\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz"
Nov 03 19:53:48.866430 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:48.840089    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727282fc-9c33-4102-96c1-de9b84f0d5d9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97\" (UID: \"727282fc-9c33-4102-96c1-de9b84f0d5d9\") " pod="openshift-ingress/payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97"
Nov 03 19:53:48.866430 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:48.840908    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7510db95-54ae-4fff-9a15-d54f80e27119-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz\" (UID: \"7510db95-54ae-4fff-9a15-d54f80e27119\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz"
Nov 03 19:53:48.901227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:48.900485    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7j7t\" (UniqueName: \"kubernetes.io/projected/7510db95-54ae-4fff-9a15-d54f80e27119-kube-api-access-c7j7t\") pod \"payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz\" (UID: \"7510db95-54ae-4fff-9a15-d54f80e27119\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz"
Nov 03 19:53:48.931232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:48.916162    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2jxf\" (UniqueName: \"kubernetes.io/projected/727282fc-9c33-4102-96c1-de9b84f0d5d9-kube-api-access-j2jxf\") pod \"payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97\" (UID: \"727282fc-9c33-4102-96c1-de9b84f0d5d9\") " pod="openshift-ingress/payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97"
Nov 03 19:53:49.066883 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:49.066844    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97"
Nov 03 19:53:49.167176 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:49.167113    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz"
Nov 03 19:53:49.406741 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:49.406711    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97"]
Nov 03 19:53:49.502159 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:49.502134    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz"]
Nov 03 19:53:49.511805 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:53:49.511775    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7510db95_54ae_4fff_9a15_d54f80e27119.slice/crio-05be33aa4549a1e9d07bb6fa518319b3503c7b9969c388f0d2c822f529f0294c WatchSource:0}: Error finding container 05be33aa4549a1e9d07bb6fa518319b3503c7b9969c388f0d2c822f529f0294c: Status 404 returned error can't find the container with id 05be33aa4549a1e9d07bb6fa518319b3503c7b9969c388f0d2c822f529f0294c
Nov 03 19:53:49.968773 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:49.968668    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz" event={"ID":"7510db95-54ae-4fff-9a15-d54f80e27119","Type":"ContainerStarted","Data":"da510dab80149c4e4d1dbcdffb34043327a995c8d630baa06a334d91e1a565e2"}
Nov 03 19:53:49.969385 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:49.969355    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz" event={"ID":"7510db95-54ae-4fff-9a15-d54f80e27119","Type":"ContainerStarted","Data":"05be33aa4549a1e9d07bb6fa518319b3503c7b9969c388f0d2c822f529f0294c"}
Nov 03 19:53:49.969542 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:49.969433    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz"
Nov 03 19:53:49.970678 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:49.970651    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97" event={"ID":"727282fc-9c33-4102-96c1-de9b84f0d5d9","Type":"ContainerStarted","Data":"3cf93d91c81b21eeb34c8dde1f5b8edbf918b3e1387b0f1c4b2ea1490a745e37"}
Nov 03 19:53:49.970808 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:49.970717    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97" event={"ID":"727282fc-9c33-4102-96c1-de9b84f0d5d9","Type":"ContainerStarted","Data":"f65642c3bfba9eef91e7d59bb02ab27f8d5aa697da594e5b8b714387239c841f"}
Nov 03 19:53:49.970808 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:49.970750    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97"
Nov 03 19:53:49.989743 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:49.989691    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz" podStartSLOduration=1.989675799 podStartE2EDuration="1.989675799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:49.986131185 +0000 UTC m=+978.758424204" watchObservedRunningTime="2026-11-03 19:53:49.989675799 +0000 UTC m=+978.761968817"
Nov 03 19:53:50.004739 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:50.004681    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97" podStartSLOduration=2.004665908 podStartE2EDuration="2.004665908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:50.004301903 +0000 UTC m=+978.776594921" watchObservedRunningTime="2026-11-03 19:53:50.004665908 +0000 UTC m=+978.776958925"
Nov 03 19:53:51.906374 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:51.906316    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:51.908690 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:51.907840    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:51.909505 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:51.907869    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 19:53:51.991230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:51.988332    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:52.420232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:52.415107    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 19:53:52.492521 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:52.492473    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:52.610287 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:52.609720    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 19:53:52.915141 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:52.915016    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:52.916122 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:52.916094    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 19:53:52.937229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:52.927558    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:53:52.937229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:52.927630    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:53:53.032954 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:53.032898    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:53:53.037508 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:53.037480    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:53:53.287938 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:53.287907    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 19:53:53.288371 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:53.288241    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 19:53:53.357099 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:53.357069    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 19:53:56.038820 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:56.038782    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8"]
Nov 03 19:53:56.044803 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:56.044779    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8"
Nov 03 19:53:56.050399 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:56.050376    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-cc7720-serving-cert\""
Nov 03 19:53:56.057705 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:56.057663    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8"]
Nov 03 19:53:56.216253 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:56.216188    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f52e2f-1d91-4822-8b8f-05975029d9c2-openshift-service-ca\") pod \"maas-api-e2e-api-a-cc7720-578688b864-hl5j8\" (UID: \"74f52e2f-1d91-4822-8b8f-05975029d9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8"
Nov 03 19:53:56.223000 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:56.222966    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74f52e2f-1d91-4822-8b8f-05975029d9c2-maas-api-tls\") pod \"maas-api-e2e-api-a-cc7720-578688b864-hl5j8\" (UID: \"74f52e2f-1d91-4822-8b8f-05975029d9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8"
Nov 03 19:53:56.223329 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:56.223299    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5chs\" (UniqueName: \"kubernetes.io/projected/74f52e2f-1d91-4822-8b8f-05975029d9c2-kube-api-access-m5chs\") pod \"maas-api-e2e-api-a-cc7720-578688b864-hl5j8\" (UID: \"74f52e2f-1d91-4822-8b8f-05975029d9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8"
Nov 03 19:53:56.230236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:56.223750    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74f52e2f-1d91-4822-8b8f-05975029d9c2-metrics-certs\") pod \"maas-api-e2e-api-a-cc7720-578688b864-hl5j8\" (UID: \"74f52e2f-1d91-4822-8b8f-05975029d9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8"
Nov 03 19:53:56.332750 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:56.332663    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f52e2f-1d91-4822-8b8f-05975029d9c2-openshift-service-ca\") pod \"maas-api-e2e-api-a-cc7720-578688b864-hl5j8\" (UID: \"74f52e2f-1d91-4822-8b8f-05975029d9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8"
Nov 03 19:53:56.332926 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:56.332767    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74f52e2f-1d91-4822-8b8f-05975029d9c2-maas-api-tls\") pod \"maas-api-e2e-api-a-cc7720-578688b864-hl5j8\" (UID: \"74f52e2f-1d91-4822-8b8f-05975029d9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8"
Nov 03 19:53:56.332926 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:56.332805    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5chs\" (UniqueName: \"kubernetes.io/projected/74f52e2f-1d91-4822-8b8f-05975029d9c2-kube-api-access-m5chs\") pod \"maas-api-e2e-api-a-cc7720-578688b864-hl5j8\" (UID: \"74f52e2f-1d91-4822-8b8f-05975029d9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8"
Nov 03 19:53:56.333020 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:56.332977    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74f52e2f-1d91-4822-8b8f-05975029d9c2-metrics-certs\") pod \"maas-api-e2e-api-a-cc7720-578688b864-hl5j8\" (UID: \"74f52e2f-1d91-4822-8b8f-05975029d9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8"
Nov 03 19:53:56.333514 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:56.333450    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f52e2f-1d91-4822-8b8f-05975029d9c2-openshift-service-ca\") pod \"maas-api-e2e-api-a-cc7720-578688b864-hl5j8\" (UID: \"74f52e2f-1d91-4822-8b8f-05975029d9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8"
Nov 03 19:53:56.335501 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:56.335476    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74f52e2f-1d91-4822-8b8f-05975029d9c2-metrics-certs\") pod \"maas-api-e2e-api-a-cc7720-578688b864-hl5j8\" (UID: \"74f52e2f-1d91-4822-8b8f-05975029d9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8"
Nov 03 19:53:56.335607 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:56.335591    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74f52e2f-1d91-4822-8b8f-05975029d9c2-maas-api-tls\") pod \"maas-api-e2e-api-a-cc7720-578688b864-hl5j8\" (UID: \"74f52e2f-1d91-4822-8b8f-05975029d9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8"
Nov 03 19:53:56.348048 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:56.348009    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5chs\" (UniqueName: \"kubernetes.io/projected/74f52e2f-1d91-4822-8b8f-05975029d9c2-kube-api-access-m5chs\") pod \"maas-api-e2e-api-a-cc7720-578688b864-hl5j8\" (UID: \"74f52e2f-1d91-4822-8b8f-05975029d9c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8"
Nov 03 19:53:56.378850 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:56.378823    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8"
Nov 03 19:53:56.661645 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:56.661615    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8"]
Nov 03 19:53:56.665997 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:56.665973    5591 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:53:57.286779 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:57.285314    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8" event={"ID":"74f52e2f-1d91-4822-8b8f-05975029d9c2","Type":"ContainerStarted","Data":"3b996f88497e6dc4cd7a8ef21af2cc29093333185e25b1e137941fd26ea167d0"}
Nov 03 19:53:57.286779 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:57.285395    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8" event={"ID":"74f52e2f-1d91-4822-8b8f-05975029d9c2","Type":"ContainerStarted","Data":"2cdf451810727d95048a2646e9504145ab74b4b5a7aaf935e297605a338371fd"}
Nov 03 19:53:57.286779 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:53:57.286717    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8"
Nov 03 19:53:58.724386 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:53:58.724345    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:54:01.062029 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:01.061998    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97"
Nov 03 19:54:01.073549 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:01.073522    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz"
Nov 03 19:54:01.094679 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:01.094603    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8" podStartSLOduration=4.637460818 podStartE2EDuration="5.094577751s" totalImagesPullingTime="457.116933ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:57.354584868 +0000 UTC m=+986.126877887" watchObservedRunningTime="2026-11-03 19:54:01.094577751 +0000 UTC m=+989.866870773"
Nov 03 19:54:04.305763 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:04.305735    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8"
Nov 03 19:54:12.243980 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:12.243901    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:54:22.637459 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:22.637421    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv"]
Nov 03 19:54:22.642450 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:22.642428    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv"
Nov 03 19:54:22.650048 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:22.650020    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ad865c\""
Nov 03 19:54:22.653973 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:22.653950    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv"]
Nov 03 19:54:22.691189 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:22.691158    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx"]
Nov 03 19:54:22.696034 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:22.696008    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx"
Nov 03 19:54:22.707261 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:22.707241    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ad865c-dockercfg-6qf8r\""
Nov 03 19:54:22.709280 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:22.709185    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx"]
Nov 03 19:54:22.905281 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:22.889368    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnc96\" (UniqueName: \"kubernetes.io/projected/37bd5a46-72a3-4db6-b3c6-2e666bb257b5-kube-api-access-fnc96\") pod \"payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv\" (UID: \"37bd5a46-72a3-4db6-b3c6-2e666bb257b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv"
Nov 03 19:54:22.905281 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:22.902602    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/607277df-ce3e-4124-b752-1256b14e69f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx\" (UID: \"607277df-ce3e-4124-b752-1256b14e69f4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx"
Nov 03 19:54:22.905281 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:22.902689    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37bd5a46-72a3-4db6-b3c6-2e666bb257b5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv\" (UID: \"37bd5a46-72a3-4db6-b3c6-2e666bb257b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv"
Nov 03 19:54:22.921539 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:22.921493    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6prx\" (UniqueName: \"kubernetes.io/projected/607277df-ce3e-4124-b752-1256b14e69f4-kube-api-access-b6prx\") pod \"payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx\" (UID: \"607277df-ce3e-4124-b752-1256b14e69f4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx"
Nov 03 19:54:23.024166 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:23.024124    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnc96\" (UniqueName: \"kubernetes.io/projected/37bd5a46-72a3-4db6-b3c6-2e666bb257b5-kube-api-access-fnc96\") pod \"payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv\" (UID: \"37bd5a46-72a3-4db6-b3c6-2e666bb257b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv"
Nov 03 19:54:23.024926 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:23.024898    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/607277df-ce3e-4124-b752-1256b14e69f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx\" (UID: \"607277df-ce3e-4124-b752-1256b14e69f4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx"
Nov 03 19:54:23.026110 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:23.026001    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/607277df-ce3e-4124-b752-1256b14e69f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx\" (UID: \"607277df-ce3e-4124-b752-1256b14e69f4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx"
Nov 03 19:54:23.026395 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:23.026377    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37bd5a46-72a3-4db6-b3c6-2e666bb257b5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv\" (UID: \"37bd5a46-72a3-4db6-b3c6-2e666bb257b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv"
Nov 03 19:54:23.027312 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:23.027294    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6prx\" (UniqueName: \"kubernetes.io/projected/607277df-ce3e-4124-b752-1256b14e69f4-kube-api-access-b6prx\") pod \"payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx\" (UID: \"607277df-ce3e-4124-b752-1256b14e69f4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx"
Nov 03 19:54:23.036834 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:23.036804    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37bd5a46-72a3-4db6-b3c6-2e666bb257b5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv\" (UID: \"37bd5a46-72a3-4db6-b3c6-2e666bb257b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv"
Nov 03 19:54:23.044013 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:23.043988    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6prx\" (UniqueName: \"kubernetes.io/projected/607277df-ce3e-4124-b752-1256b14e69f4-kube-api-access-b6prx\") pod \"payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx\" (UID: \"607277df-ce3e-4124-b752-1256b14e69f4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx"
Nov 03 19:54:23.070517 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:23.070486    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnc96\" (UniqueName: \"kubernetes.io/projected/37bd5a46-72a3-4db6-b3c6-2e666bb257b5-kube-api-access-fnc96\") pod \"payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv\" (UID: \"37bd5a46-72a3-4db6-b3c6-2e666bb257b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv"
Nov 03 19:54:23.260408 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:23.260321    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv"
Nov 03 19:54:23.313690 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:23.313658    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx"
Nov 03 19:54:23.579588 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:23.573572    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv"]
Nov 03 19:54:23.674235 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:23.672306    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx"]
Nov 03 19:54:23.977090 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:23.974325    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx" event={"ID":"607277df-ce3e-4124-b752-1256b14e69f4","Type":"ContainerStarted","Data":"c08e6cdd21f0318644da0b7878941ad0fc7e66ff5d4c2807d8e2fa835bbde0b8"}
Nov 03 19:54:23.978126 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:23.977426    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv" event={"ID":"37bd5a46-72a3-4db6-b3c6-2e666bb257b5","Type":"ContainerStarted","Data":"ef59c503737a42cda6a7968f9826ba3c25d75bb1ef875b20305bba2501edced0"}
Nov 03 19:54:24.986685 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:24.986605    5591 generic.go:356] "Generic (PLEG): container finished" podID="fc0e2592-7378-4f51-99b4-6b4f06ecc61c" containerID="44754f7322be589979ca35b6e1215c8814e1921d6996837dc6c14aead87b490c" exitCode=0
Nov 03 19:54:24.986685 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:24.986676    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn" event={"ID":"fc0e2592-7378-4f51-99b4-6b4f06ecc61c","Type":"ContainerDied","Data":"44754f7322be589979ca35b6e1215c8814e1921d6996837dc6c14aead87b490c"}
Nov 03 19:54:24.987303 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:24.987173    5591 scope.go:122] "RemoveContainer" containerID="44754f7322be589979ca35b6e1215c8814e1921d6996837dc6c14aead87b490c"
Nov 03 19:54:24.988375 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:24.988352    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx" event={"ID":"607277df-ce3e-4124-b752-1256b14e69f4","Type":"ContainerStarted","Data":"69233cf99d220b67859f91f890f094077a8feb900cd7e9f98337874b4af4e958"}
Nov 03 19:54:24.988485 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:24.988415    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx"
Nov 03 19:54:24.989030 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:24.988992    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:54:24.989144 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:24.989071    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:54:24.989144 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:24.989119    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:54:24.990361 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:24.990336    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv" event={"ID":"37bd5a46-72a3-4db6-b3c6-2e666bb257b5","Type":"ContainerStarted","Data":"cb0c2dce1ffa5d501acb888a340ef0b617d802633c101c34577bf77aeb0eccfa"}
Nov 03 19:54:24.990475 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:24.990383    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv"
Nov 03 19:54:25.036077 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:25.036018    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx" podStartSLOduration=3.035997411 podStartE2EDuration="3.035997411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:25.034942122 +0000 UTC m=+1013.807235138" watchObservedRunningTime="2026-11-03 19:54:25.035997411 +0000 UTC m=+1013.808290430"
Nov 03 19:54:25.066314 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:25.066266    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv" podStartSLOduration=3.066249485 podStartE2EDuration="3.066249485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:25.065401097 +0000 UTC m=+1013.837694117" watchObservedRunningTime="2026-11-03 19:54:25.066249485 +0000 UTC m=+1013.838542503"
Nov 03 19:54:26.012004 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:26.010237    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn" event={"ID":"fc0e2592-7378-4f51-99b4-6b4f06ecc61c","Type":"ContainerStarted","Data":"570929fa37a1956fdc84f00b4d511dc6b278759febce6527110fa1eed9a78b9c"}
Nov 03 19:54:26.683638 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:26.683606    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:54:26.688495 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:26.688459    5591 patch_prober.go:29] interesting pod/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 19:54:26.688785 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:26.688740    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn" podUID="fc0e2592-7378-4f51-99b4-6b4f06ecc61c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 19:54:27.684326 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:27.684292    5591 patch_prober.go:29] interesting pod/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 19:54:27.684961 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:27.684930    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn" podUID="fc0e2592-7378-4f51-99b4-6b4f06ecc61c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 19:54:28.685699 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:28.685654    5591 patch_prober.go:29] interesting pod/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 19:54:28.708231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:28.688359    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn" podUID="fc0e2592-7378-4f51-99b4-6b4f06ecc61c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 19:54:28.745155 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:54:28.745120    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:54:28.806250 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:28.796779    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb"]
Nov 03 19:54:28.813537 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:28.813360    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb"]
Nov 03 19:54:28.813537 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:28.813452    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb"
Nov 03 19:54:28.820783 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:28.817328    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz"]
Nov 03 19:54:28.824806 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:28.824678    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f20fc0-dockercfg-jz92c\""
Nov 03 19:54:28.824903 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:28.824812    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f20fc0\""
Nov 03 19:54:28.831009 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:28.830982    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz"]
Nov 03 19:54:28.831269 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:28.831081    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz"
Nov 03 19:54:28.866163 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:28.866124    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c84mq\" (UniqueName: \"kubernetes.io/projected/aee9f77c-8ccd-4e61-bbb2-5abab68d460b-kube-api-access-c84mq\") pod \"payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz\" (UID: \"aee9f77c-8ccd-4e61-bbb2-5abab68d460b\") " pod="openshift-ingress/payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz"
Nov 03 19:54:28.866434 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:28.866232    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efa31e2a-d92f-4a8e-98d0-451b78d41880-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb\" (UID: \"efa31e2a-d92f-4a8e-98d0-451b78d41880\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb"
Nov 03 19:54:28.866434 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:28.866301    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmbmp\" (UniqueName: \"kubernetes.io/projected/efa31e2a-d92f-4a8e-98d0-451b78d41880-kube-api-access-fmbmp\") pod \"payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb\" (UID: \"efa31e2a-d92f-4a8e-98d0-451b78d41880\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb"
Nov 03 19:54:28.866643 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:28.866562    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aee9f77c-8ccd-4e61-bbb2-5abab68d460b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz\" (UID: \"aee9f77c-8ccd-4e61-bbb2-5abab68d460b\") " pod="openshift-ingress/payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz"
Nov 03 19:54:28.993290 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:28.978410    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c84mq\" (UniqueName: \"kubernetes.io/projected/aee9f77c-8ccd-4e61-bbb2-5abab68d460b-kube-api-access-c84mq\") pod \"payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz\" (UID: \"aee9f77c-8ccd-4e61-bbb2-5abab68d460b\") " pod="openshift-ingress/payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz"
Nov 03 19:54:28.993290 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:28.978514    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efa31e2a-d92f-4a8e-98d0-451b78d41880-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb\" (UID: \"efa31e2a-d92f-4a8e-98d0-451b78d41880\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb"
Nov 03 19:54:28.993290 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:28.978584    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmbmp\" (UniqueName: \"kubernetes.io/projected/efa31e2a-d92f-4a8e-98d0-451b78d41880-kube-api-access-fmbmp\") pod \"payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb\" (UID: \"efa31e2a-d92f-4a8e-98d0-451b78d41880\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb"
Nov 03 19:54:28.993290 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:28.978836    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aee9f77c-8ccd-4e61-bbb2-5abab68d460b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz\" (UID: \"aee9f77c-8ccd-4e61-bbb2-5abab68d460b\") " pod="openshift-ingress/payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz"
Nov 03 19:54:28.993290 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:28.980058    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aee9f77c-8ccd-4e61-bbb2-5abab68d460b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz\" (UID: \"aee9f77c-8ccd-4e61-bbb2-5abab68d460b\") " pod="openshift-ingress/payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz"
Nov 03 19:54:28.993290 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:28.981157    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efa31e2a-d92f-4a8e-98d0-451b78d41880-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb\" (UID: \"efa31e2a-d92f-4a8e-98d0-451b78d41880\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb"
Nov 03 19:54:29.144246 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:29.134502    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c84mq\" (UniqueName: \"kubernetes.io/projected/aee9f77c-8ccd-4e61-bbb2-5abab68d460b-kube-api-access-c84mq\") pod \"payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz\" (UID: \"aee9f77c-8ccd-4e61-bbb2-5abab68d460b\") " pod="openshift-ingress/payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz"
Nov 03 19:54:29.152669 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:29.152622    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz"
Nov 03 19:54:29.244326 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:29.236594    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmbmp\" (UniqueName: \"kubernetes.io/projected/efa31e2a-d92f-4a8e-98d0-451b78d41880-kube-api-access-fmbmp\") pod \"payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb\" (UID: \"efa31e2a-d92f-4a8e-98d0-451b78d41880\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb"
Nov 03 19:54:29.434086 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:29.434055    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb"
Nov 03 19:54:29.459095 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:29.459063    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d"]
Nov 03 19:54:29.466025 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:29.465946    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d"
Nov 03 19:54:29.473235 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:29.470587    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ad865c-serving-cert\""
Nov 03 19:54:29.500276 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:29.489265    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d"]
Nov 03 19:54:29.500276 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:29.499081    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbcpk\" (UniqueName: \"kubernetes.io/projected/da8f2c4a-8edd-450c-b456-dc6ed51dc851-kube-api-access-hbcpk\") pod \"maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d\" (UID: \"da8f2c4a-8edd-450c-b456-dc6ed51dc851\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d"
Nov 03 19:54:29.500276 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:29.499191    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8f2c4a-8edd-450c-b456-dc6ed51dc851-openshift-service-ca\") pod \"maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d\" (UID: \"da8f2c4a-8edd-450c-b456-dc6ed51dc851\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d"
Nov 03 19:54:29.500276 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:29.499267    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da8f2c4a-8edd-450c-b456-dc6ed51dc851-maas-api-tls\") pod \"maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d\" (UID: \"da8f2c4a-8edd-450c-b456-dc6ed51dc851\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d"
Nov 03 19:54:29.500276 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:29.499291    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da8f2c4a-8edd-450c-b456-dc6ed51dc851-metrics-certs\") pod \"maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d\" (UID: \"da8f2c4a-8edd-450c-b456-dc6ed51dc851\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d"
Nov 03 19:54:29.621559 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:29.600387    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8f2c4a-8edd-450c-b456-dc6ed51dc851-openshift-service-ca\") pod \"maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d\" (UID: \"da8f2c4a-8edd-450c-b456-dc6ed51dc851\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d"
Nov 03 19:54:29.621559 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:29.600443    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da8f2c4a-8edd-450c-b456-dc6ed51dc851-maas-api-tls\") pod \"maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d\" (UID: \"da8f2c4a-8edd-450c-b456-dc6ed51dc851\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d"
Nov 03 19:54:29.621559 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:29.600469    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da8f2c4a-8edd-450c-b456-dc6ed51dc851-metrics-certs\") pod \"maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d\" (UID: \"da8f2c4a-8edd-450c-b456-dc6ed51dc851\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d"
Nov 03 19:54:29.621559 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:29.600675    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbcpk\" (UniqueName: \"kubernetes.io/projected/da8f2c4a-8edd-450c-b456-dc6ed51dc851-kube-api-access-hbcpk\") pod \"maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d\" (UID: \"da8f2c4a-8edd-450c-b456-dc6ed51dc851\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d"
Nov 03 19:54:29.621559 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:29.601718    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8f2c4a-8edd-450c-b456-dc6ed51dc851-openshift-service-ca\") pod \"maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d\" (UID: \"da8f2c4a-8edd-450c-b456-dc6ed51dc851\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d"
Nov 03 19:54:29.621559 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:29.612700    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da8f2c4a-8edd-450c-b456-dc6ed51dc851-metrics-certs\") pod \"maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d\" (UID: \"da8f2c4a-8edd-450c-b456-dc6ed51dc851\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d"
Nov 03 19:54:29.621559 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:29.615294    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz"]
Nov 03 19:54:29.621559 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:29.617673    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da8f2c4a-8edd-450c-b456-dc6ed51dc851-maas-api-tls\") pod \"maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d\" (UID: \"da8f2c4a-8edd-450c-b456-dc6ed51dc851\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d"
Nov 03 19:54:29.621559 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:29.618397    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbcpk\" (UniqueName: \"kubernetes.io/projected/da8f2c4a-8edd-450c-b456-dc6ed51dc851-kube-api-access-hbcpk\") pod \"maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d\" (UID: \"da8f2c4a-8edd-450c-b456-dc6ed51dc851\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d"
Nov 03 19:54:29.683551 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:29.683509    5591 patch_prober.go:29] interesting pod/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 19:54:29.687235 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:29.683843    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn" podUID="fc0e2592-7378-4f51-99b4-6b4f06ecc61c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 19:54:29.843777 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:29.843746    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d"
Nov 03 19:54:29.990829 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:29.990346    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb"]
Nov 03 19:54:30.149229 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:54:30.145326    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa31e2a_d92f_4a8e_98d0_451b78d41880.slice/crio-2465a376f413daaf47e009112c2360b8773ab5fec800f8df1c902c422017fd0b WatchSource:0}: Error finding container 2465a376f413daaf47e009112c2360b8773ab5fec800f8df1c902c422017fd0b: Status 404 returned error can't find the container with id 2465a376f413daaf47e009112c2360b8773ab5fec800f8df1c902c422017fd0b
Nov 03 19:54:30.257108 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:30.234314    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d"]
Nov 03 19:54:30.452386 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:30.452350    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb" event={"ID":"efa31e2a-d92f-4a8e-98d0-451b78d41880","Type":"ContainerStarted","Data":"2465a376f413daaf47e009112c2360b8773ab5fec800f8df1c902c422017fd0b"}
Nov 03 19:54:30.454564 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:30.454533    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz" event={"ID":"aee9f77c-8ccd-4e61-bbb2-5abab68d460b","Type":"ContainerStarted","Data":"693251472c3f25355670145542e426f73810a26a2801198712566f1cb842d303"}
Nov 03 19:54:30.454683 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:30.454590    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz" event={"ID":"aee9f77c-8ccd-4e61-bbb2-5abab68d460b","Type":"ContainerStarted","Data":"e4c8518db89433e0cfe54a6f82ffd6d173689fc746a4df05011010d41207de1a"}
Nov 03 19:54:30.455805 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:30.455784    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz"
Nov 03 19:54:30.512239 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:30.490732    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz" podStartSLOduration=2.490709419 podStartE2EDuration="2.490709419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:30.474059631 +0000 UTC m=+1019.246352649" watchObservedRunningTime="2026-11-03 19:54:30.490709419 +0000 UTC m=+1019.263002437"
Nov 03 19:54:30.701527 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:30.701381    5591 patch_prober.go:29] interesting pod/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 03 19:54:30.701527 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:30.701451    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn" podUID="fc0e2592-7378-4f51-99b4-6b4f06ecc61c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 03 19:54:30.933229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:30.929777    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:54:31.465514 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:31.464999    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d" event={"ID":"da8f2c4a-8edd-450c-b456-dc6ed51dc851","Type":"ContainerStarted","Data":"89a89d30fd6abf625ae7ec8a4fc7b4169df17aff33d8bb9cd12b4585d4c15813"}
Nov 03 19:54:31.465514 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:31.465063    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d" event={"ID":"da8f2c4a-8edd-450c-b456-dc6ed51dc851","Type":"ContainerStarted","Data":"74823096ff78e01217c84aece4c3d75d57028541cc0a840505a876f88e75d5e2"}
Nov 03 19:54:31.465825 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:31.465805    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d"
Nov 03 19:54:31.467357 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:31.467329    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb" event={"ID":"efa31e2a-d92f-4a8e-98d0-451b78d41880","Type":"ContainerStarted","Data":"a3a417fedf8c60625efdfd660dc319ef3a99f3194f5a4e627e8c79c7a94c6d5f"}
Nov 03 19:54:31.467503 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:31.467489    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb"
Nov 03 19:54:31.491228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:31.489128    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d" podStartSLOduration=1.816708412 podStartE2EDuration="2.489107725s" totalImagesPullingTime="672.399313ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:31.487522461 +0000 UTC m=+1020.259815480" watchObservedRunningTime="2026-11-03 19:54:31.489107725 +0000 UTC m=+1020.261400741"
Nov 03 19:54:31.507783 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:31.507733    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb" podStartSLOduration=3.507718268 podStartE2EDuration="3.507718268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:31.505135633 +0000 UTC m=+1020.277428663" watchObservedRunningTime="2026-11-03 19:54:31.507718268 +0000 UTC m=+1020.280011311"
Nov 03 19:54:31.654231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:31.649069    5591 scope.go:122] "RemoveContainer" containerID="fcaa32b0cc3823c4464c0d4c473e0c57742a070f379daf7daf8630a99cebcf3f"
Nov 03 19:54:31.872537 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:31.872499    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:54:31.874157 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:31.874130    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:54:35.136901 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:35.136863    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78"]
Nov 03 19:54:35.144395 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:35.144361    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78"
Nov 03 19:54:35.148345 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:35.148320    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f20fc0-serving-cert\""
Nov 03 19:54:35.157083 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:35.157058    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78"]
Nov 03 19:54:35.176653 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:35.176620    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af7d8f35-2a5e-4998-a2c9-7637747b374a-openshift-service-ca\") pod \"maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78\" (UID: \"af7d8f35-2a5e-4998-a2c9-7637747b374a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78"
Nov 03 19:54:35.176902 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:35.176877    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af7d8f35-2a5e-4998-a2c9-7637747b374a-metrics-certs\") pod \"maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78\" (UID: \"af7d8f35-2a5e-4998-a2c9-7637747b374a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78"
Nov 03 19:54:35.177014 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:35.176995    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af7d8f35-2a5e-4998-a2c9-7637747b374a-maas-api-tls\") pod \"maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78\" (UID: \"af7d8f35-2a5e-4998-a2c9-7637747b374a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78"
Nov 03 19:54:35.177076 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:35.177042    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvmr4\" (UniqueName: \"kubernetes.io/projected/af7d8f35-2a5e-4998-a2c9-7637747b374a-kube-api-access-lvmr4\") pod \"maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78\" (UID: \"af7d8f35-2a5e-4998-a2c9-7637747b374a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78"
Nov 03 19:54:35.300232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:35.284486    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af7d8f35-2a5e-4998-a2c9-7637747b374a-metrics-certs\") pod \"maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78\" (UID: \"af7d8f35-2a5e-4998-a2c9-7637747b374a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78"
Nov 03 19:54:35.300232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:35.284652    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af7d8f35-2a5e-4998-a2c9-7637747b374a-maas-api-tls\") pod \"maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78\" (UID: \"af7d8f35-2a5e-4998-a2c9-7637747b374a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78"
Nov 03 19:54:35.300232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:35.284713    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvmr4\" (UniqueName: \"kubernetes.io/projected/af7d8f35-2a5e-4998-a2c9-7637747b374a-kube-api-access-lvmr4\") pod \"maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78\" (UID: \"af7d8f35-2a5e-4998-a2c9-7637747b374a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78"
Nov 03 19:54:35.300232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:35.284826    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af7d8f35-2a5e-4998-a2c9-7637747b374a-openshift-service-ca\") pod \"maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78\" (UID: \"af7d8f35-2a5e-4998-a2c9-7637747b374a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78"
Nov 03 19:54:35.300232 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:54:35.290320    5591 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-serving-cert: secret "maas-api-e2e-api-b-f20fc0-serving-cert" not found
Nov 03 19:54:35.300232 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:54:35.290411    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af7d8f35-2a5e-4998-a2c9-7637747b374a-maas-api-tls podName:af7d8f35-2a5e-4998-a2c9-7637747b374a nodeName:}" failed. No retries permitted until 2026-11-03 19:54:35.790390775 +0000 UTC m=+1024.562683788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/af7d8f35-2a5e-4998-a2c9-7637747b374a-maas-api-tls") pod "maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78" (UID: "af7d8f35-2a5e-4998-a2c9-7637747b374a") : secret "maas-api-e2e-api-b-f20fc0-serving-cert" not found
Nov 03 19:54:35.300232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:35.290798    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af7d8f35-2a5e-4998-a2c9-7637747b374a-openshift-service-ca\") pod \"maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78\" (UID: \"af7d8f35-2a5e-4998-a2c9-7637747b374a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78"
Nov 03 19:54:35.304224 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:35.301429    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af7d8f35-2a5e-4998-a2c9-7637747b374a-metrics-certs\") pod \"maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78\" (UID: \"af7d8f35-2a5e-4998-a2c9-7637747b374a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78"
Nov 03 19:54:35.311228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:35.306704    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvmr4\" (UniqueName: \"kubernetes.io/projected/af7d8f35-2a5e-4998-a2c9-7637747b374a-kube-api-access-lvmr4\") pod \"maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78\" (UID: \"af7d8f35-2a5e-4998-a2c9-7637747b374a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78"
Nov 03 19:54:35.804229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:35.803656    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af7d8f35-2a5e-4998-a2c9-7637747b374a-maas-api-tls\") pod \"maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78\" (UID: \"af7d8f35-2a5e-4998-a2c9-7637747b374a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78"
Nov 03 19:54:35.844231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:35.837775    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af7d8f35-2a5e-4998-a2c9-7637747b374a-maas-api-tls\") pod \"maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78\" (UID: \"af7d8f35-2a5e-4998-a2c9-7637747b374a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78"
Nov 03 19:54:36.013644 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:36.013616    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx"
Nov 03 19:54:36.013844 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:36.013691    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv"
Nov 03 19:54:36.077456 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:36.071414    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78"
Nov 03 19:54:36.454352 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:36.444277    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78"]
Nov 03 19:54:36.532008 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:36.531969    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78" event={"ID":"af7d8f35-2a5e-4998-a2c9-7637747b374a","Type":"ContainerStarted","Data":"855563b54f0172beefa0b8e0e2a440ec296b4029fb376d51cfa8f93c59992540"}
Nov 03 19:54:37.480479 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:37.480452    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d"
Nov 03 19:54:37.543739 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:37.543706    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78" event={"ID":"af7d8f35-2a5e-4998-a2c9-7637747b374a","Type":"ContainerStarted","Data":"d8c94cf025d7b83d5e45ca3acee5ad3ee56f8ff0a710696fe8c3f42253c31685"}
Nov 03 19:54:37.544000 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:37.543980    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78"
Nov 03 19:54:37.573868 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:37.573807    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78" podStartSLOduration=2.146068375 podStartE2EDuration="2.573789065s" totalImagesPullingTime="427.72069ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:54:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:54:37.571430994 +0000 UTC m=+1026.343724018" watchObservedRunningTime="2026-11-03 19:54:37.573789065 +0000 UTC m=+1026.346082084"
Nov 03 19:54:42.508330 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:42.506917    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb"
Nov 03 19:54:42.509138 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:42.509115    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz"
Nov 03 19:54:43.558100 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:43.558075    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78"
Nov 03 19:54:46.039861 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:46.039826    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:54:48.619237 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:48.617218    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:48.941320 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:48.884243    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:49.058319 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:49.058280    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-9cvzn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:54:49.058594 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:49.058565    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn" podUID="3a0f28bc-d05f-45bb-8431-3aaf73529510" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:49.185084 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:49.185039    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded"
Nov 03 19:54:49.389365 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:49.389318    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:49.422585 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:49.422541    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:49.472342 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:49.472294    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:49.624331 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:49.624284    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:49.625334 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:49.625306    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:54:50.037801 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:50.037770    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 19:54:50.269417 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:50.269353    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-zst5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:54:50.269623 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:50.269431    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d" podUID="e8c03ef4-fb42-4a08-b02d-98165eb9b203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:50.393602 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:50.393503    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-t8m7q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:54:50.393602 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:50.393572    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q" podUID="68b82f0c-e037-4366-891b-1ee898b2cba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:54:54.824494 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:54.824459    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97"]
Nov 03 19:54:54.841300 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:54.841269    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz"]
Nov 03 19:54:56.473228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:56.468688    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8"]
Nov 03 19:54:56.473228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:56.468981    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8" podUID="74f52e2f-1d91-4822-8b8f-05975029d9c2" containerName="maas-api" containerID="cri-o://3b996f88497e6dc4cd7a8ef21af2cc29093333185e25b1e137941fd26ea167d0" gracePeriod=30
Nov 03 19:54:56.933282 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:56.933183    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8"
Nov 03 19:54:57.081555 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:57.070878    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74f52e2f-1d91-4822-8b8f-05975029d9c2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f52e2f-1d91-4822-8b8f-05975029d9c2-openshift-service-ca\") pod \"74f52e2f-1d91-4822-8b8f-05975029d9c2\" (UID: \"74f52e2f-1d91-4822-8b8f-05975029d9c2\") "
Nov 03 19:54:57.082162 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:57.082137    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74f52e2f-1d91-4822-8b8f-05975029d9c2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74f52e2f-1d91-4822-8b8f-05975029d9c2-maas-api-tls\") pod \"74f52e2f-1d91-4822-8b8f-05975029d9c2\" (UID: \"74f52e2f-1d91-4822-8b8f-05975029d9c2\") "
Nov 03 19:54:57.109426 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:57.081426    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f52e2f-1d91-4822-8b8f-05975029d9c2-openshift-service-ca" pod "74f52e2f-1d91-4822-8b8f-05975029d9c2" (UID: "74f52e2f-1d91-4822-8b8f-05975029d9c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:54:57.109426 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:57.090527    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74f52e2f-1d91-4822-8b8f-05975029d9c2-maas-api-tls" pod "74f52e2f-1d91-4822-8b8f-05975029d9c2" (UID: "74f52e2f-1d91-4822-8b8f-05975029d9c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:54:57.109932 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:57.109899    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f52e2f-1d91-4822-8b8f-05975029d9c2-kube-api-access-m5chs\" (UniqueName: \"kubernetes.io/projected/74f52e2f-1d91-4822-8b8f-05975029d9c2-kube-api-access-m5chs\") pod \"74f52e2f-1d91-4822-8b8f-05975029d9c2\" (UID: \"74f52e2f-1d91-4822-8b8f-05975029d9c2\") "
Nov 03 19:54:57.110693 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:57.110647    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74f52e2f-1d91-4822-8b8f-05975029d9c2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/74f52e2f-1d91-4822-8b8f-05975029d9c2-metrics-certs\") pod \"74f52e2f-1d91-4822-8b8f-05975029d9c2\" (UID: \"74f52e2f-1d91-4822-8b8f-05975029d9c2\") "
Nov 03 19:54:57.112293 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:57.112195    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f52e2f-1d91-4822-8b8f-05975029d9c2-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:57.112487 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:57.112474    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74f52e2f-1d91-4822-8b8f-05975029d9c2-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:57.153833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:57.153790    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f52e2f-1d91-4822-8b8f-05975029d9c2-kube-api-access-m5chs" pod "74f52e2f-1d91-4822-8b8f-05975029d9c2" (UID: "74f52e2f-1d91-4822-8b8f-05975029d9c2"). InnerVolumeSpecName "kube-api-access-m5chs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:54:57.171651 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:57.169008    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74f52e2f-1d91-4822-8b8f-05975029d9c2-metrics-certs" pod "74f52e2f-1d91-4822-8b8f-05975029d9c2" (UID: "74f52e2f-1d91-4822-8b8f-05975029d9c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:54:57.207736 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:57.207696    5591 generic.go:356] "Generic (PLEG): container finished" podID="74f52e2f-1d91-4822-8b8f-05975029d9c2" containerID="3b996f88497e6dc4cd7a8ef21af2cc29093333185e25b1e137941fd26ea167d0" exitCode=0
Nov 03 19:54:57.208167 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:57.208141    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8" event={"ID":"74f52e2f-1d91-4822-8b8f-05975029d9c2","Type":"ContainerDied","Data":"3b996f88497e6dc4cd7a8ef21af2cc29093333185e25b1e137941fd26ea167d0"}
Nov 03 19:54:57.208616 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:57.208598    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8" event={"ID":"74f52e2f-1d91-4822-8b8f-05975029d9c2","Type":"ContainerDied","Data":"2cdf451810727d95048a2646e9504145ab74b4b5a7aaf935e297605a338371fd"}
Nov 03 19:54:57.208809 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:57.208795    5591 scope.go:122] "RemoveContainer" containerID="3b996f88497e6dc4cd7a8ef21af2cc29093333185e25b1e137941fd26ea167d0"
Nov 03 19:54:57.209039 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:57.208423    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8"
Nov 03 19:54:57.214127 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:57.213997    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5chs\" (UniqueName: \"kubernetes.io/projected/74f52e2f-1d91-4822-8b8f-05975029d9c2-kube-api-access-m5chs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:57.214127 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:57.214030    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74f52e2f-1d91-4822-8b8f-05975029d9c2-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:54:57.459260 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:57.452928    5591 scope.go:122] "RemoveContainer" containerID="3b996f88497e6dc4cd7a8ef21af2cc29093333185e25b1e137941fd26ea167d0"
Nov 03 19:54:57.493227 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:54:57.483381    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b996f88497e6dc4cd7a8ef21af2cc29093333185e25b1e137941fd26ea167d0\": container with ID starting with 3b996f88497e6dc4cd7a8ef21af2cc29093333185e25b1e137941fd26ea167d0 not found: ID does not exist" containerID="3b996f88497e6dc4cd7a8ef21af2cc29093333185e25b1e137941fd26ea167d0"
Nov 03 19:54:57.493227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:57.483430    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b996f88497e6dc4cd7a8ef21af2cc29093333185e25b1e137941fd26ea167d0"} err="failed to get container status \"3b996f88497e6dc4cd7a8ef21af2cc29093333185e25b1e137941fd26ea167d0\": rpc error: code = NotFound desc = could not find container \"3b996f88497e6dc4cd7a8ef21af2cc29093333185e25b1e137941fd26ea167d0\": container with ID starting with 3b996f88497e6dc4cd7a8ef21af2cc29093333185e25b1e137941fd26ea167d0 not found: ID does not exist"
Nov 03 19:54:57.514278 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:57.505016    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8"]
Nov 03 19:54:57.515157 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:57.515130    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cc7720-578688b864-hl5j8"]
Nov 03 19:54:58.726541 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:54:58.726501    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:54:59.381044 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:59.381008    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74f52e2f-1d91-4822-8b8f-05975029d9c2" path="/var/lib/kubelet/pods/74f52e2f-1d91-4822-8b8f-05975029d9c2/volumes"
Nov 03 19:54:59.825583 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:59.825541    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97" podUID="727282fc-9c33-4102-96c1-de9b84f0d5d9" containerName="payload-processing" containerID="cri-o://3cf93d91c81b21eeb34c8dde1f5b8edbf918b3e1387b0f1c4b2ea1490a745e37" gracePeriod=25
Nov 03 19:54:59.842106 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:54:59.842068    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz" podUID="7510db95-54ae-4fff-9a15-d54f80e27119" containerName="payload-pre-processing" containerID="cri-o://da510dab80149c4e4d1dbcdffb34043327a995c8d630baa06a334d91e1a565e2" gracePeriod=25
Nov 03 19:55:00.224841 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:00.224707    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97"
Nov 03 19:55:00.242887 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:00.242853    5591 generic.go:356] "Generic (PLEG): container finished" podID="7510db95-54ae-4fff-9a15-d54f80e27119" containerID="da510dab80149c4e4d1dbcdffb34043327a995c8d630baa06a334d91e1a565e2" exitCode=0
Nov 03 19:55:00.243051 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:00.242937    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz" event={"ID":"7510db95-54ae-4fff-9a15-d54f80e27119","Type":"ContainerDied","Data":"da510dab80149c4e4d1dbcdffb34043327a995c8d630baa06a334d91e1a565e2"}
Nov 03 19:55:00.255263 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:00.245150    5591 generic.go:356] "Generic (PLEG): container finished" podID="727282fc-9c33-4102-96c1-de9b84f0d5d9" containerID="3cf93d91c81b21eeb34c8dde1f5b8edbf918b3e1387b0f1c4b2ea1490a745e37" exitCode=0
Nov 03 19:55:00.255263 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:00.245353    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97" event={"ID":"727282fc-9c33-4102-96c1-de9b84f0d5d9","Type":"ContainerDied","Data":"3cf93d91c81b21eeb34c8dde1f5b8edbf918b3e1387b0f1c4b2ea1490a745e37"}
Nov 03 19:55:00.255263 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:00.245395    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97" event={"ID":"727282fc-9c33-4102-96c1-de9b84f0d5d9","Type":"ContainerDied","Data":"f65642c3bfba9eef91e7d59bb02ab27f8d5aa697da594e5b8b714387239c841f"}
Nov 03 19:55:00.255263 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:00.245416    5591 scope.go:122] "RemoveContainer" containerID="3cf93d91c81b21eeb34c8dde1f5b8edbf918b3e1387b0f1c4b2ea1490a745e37"
Nov 03 19:55:00.255263 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:00.245509    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97"
Nov 03 19:55:00.259980 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:00.259953    5591 scope.go:122] "RemoveContainer" containerID="3cf93d91c81b21eeb34c8dde1f5b8edbf918b3e1387b0f1c4b2ea1490a745e37"
Nov 03 19:55:00.260446 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:55:00.260402    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cf93d91c81b21eeb34c8dde1f5b8edbf918b3e1387b0f1c4b2ea1490a745e37\": container with ID starting with 3cf93d91c81b21eeb34c8dde1f5b8edbf918b3e1387b0f1c4b2ea1490a745e37 not found: ID does not exist" containerID="3cf93d91c81b21eeb34c8dde1f5b8edbf918b3e1387b0f1c4b2ea1490a745e37"
Nov 03 19:55:00.260812 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:00.260776    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf93d91c81b21eeb34c8dde1f5b8edbf918b3e1387b0f1c4b2ea1490a745e37"} err="failed to get container status \"3cf93d91c81b21eeb34c8dde1f5b8edbf918b3e1387b0f1c4b2ea1490a745e37\": rpc error: code = NotFound desc = could not find container \"3cf93d91c81b21eeb34c8dde1f5b8edbf918b3e1387b0f1c4b2ea1490a745e37\": container with ID starting with 3cf93d91c81b21eeb34c8dde1f5b8edbf918b3e1387b0f1c4b2ea1490a745e37 not found: ID does not exist"
Nov 03 19:55:00.300058 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:00.300027    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz"
Nov 03 19:55:00.358870 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:00.358834    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7510db95-54ae-4fff-9a15-d54f80e27119-kube-api-access-c7j7t\" (UniqueName: \"kubernetes.io/projected/7510db95-54ae-4fff-9a15-d54f80e27119-kube-api-access-c7j7t\") pod \"7510db95-54ae-4fff-9a15-d54f80e27119\" (UID: \"7510db95-54ae-4fff-9a15-d54f80e27119\") "
Nov 03 19:55:00.359076 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:00.358898    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/727282fc-9c33-4102-96c1-de9b84f0d5d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727282fc-9c33-4102-96c1-de9b84f0d5d9-plugins-config-volume\") pod \"727282fc-9c33-4102-96c1-de9b84f0d5d9\" (UID: \"727282fc-9c33-4102-96c1-de9b84f0d5d9\") "
Nov 03 19:55:00.359076 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:00.358988    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/727282fc-9c33-4102-96c1-de9b84f0d5d9-kube-api-access-j2jxf\" (UniqueName: \"kubernetes.io/projected/727282fc-9c33-4102-96c1-de9b84f0d5d9-kube-api-access-j2jxf\") pod \"727282fc-9c33-4102-96c1-de9b84f0d5d9\" (UID: \"727282fc-9c33-4102-96c1-de9b84f0d5d9\") "
Nov 03 19:55:00.359197 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:00.359112    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7510db95-54ae-4fff-9a15-d54f80e27119-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7510db95-54ae-4fff-9a15-d54f80e27119-plugins-config-volume\") pod \"7510db95-54ae-4fff-9a15-d54f80e27119\" (UID: \"7510db95-54ae-4fff-9a15-d54f80e27119\") "
Nov 03 19:55:00.360142 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:00.360101    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/727282fc-9c33-4102-96c1-de9b84f0d5d9-plugins-config-volume" pod "727282fc-9c33-4102-96c1-de9b84f0d5d9" (UID: "727282fc-9c33-4102-96c1-de9b84f0d5d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:55:00.361609 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:00.361583    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7510db95-54ae-4fff-9a15-d54f80e27119-kube-api-access-c7j7t" pod "7510db95-54ae-4fff-9a15-d54f80e27119" (UID: "7510db95-54ae-4fff-9a15-d54f80e27119"). InnerVolumeSpecName "kube-api-access-c7j7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:55:00.361870 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:00.361831    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7510db95-54ae-4fff-9a15-d54f80e27119-plugins-config-volume" pod "7510db95-54ae-4fff-9a15-d54f80e27119" (UID: "7510db95-54ae-4fff-9a15-d54f80e27119"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:55:00.362755 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:00.362731    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7j7t\" (UniqueName: \"kubernetes.io/projected/7510db95-54ae-4fff-9a15-d54f80e27119-kube-api-access-c7j7t\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:00.362847 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:00.362768    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/727282fc-9c33-4102-96c1-de9b84f0d5d9-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:00.362847 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:00.362783    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7510db95-54ae-4fff-9a15-d54f80e27119-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:00.363020 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:00.362961    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/727282fc-9c33-4102-96c1-de9b84f0d5d9-kube-api-access-j2jxf" pod "727282fc-9c33-4102-96c1-de9b84f0d5d9" (UID: "727282fc-9c33-4102-96c1-de9b84f0d5d9"). InnerVolumeSpecName "kube-api-access-j2jxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:55:00.464384 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:00.464354    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2jxf\" (UniqueName: \"kubernetes.io/projected/727282fc-9c33-4102-96c1-de9b84f0d5d9-kube-api-access-j2jxf\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:00.573733 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:00.573695    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97"]
Nov 03 19:55:00.577471 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:00.577447    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-cc7720-658f6844c8-kdz97"]
Nov 03 19:55:01.254030 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:01.253923    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz" event={"ID":"7510db95-54ae-4fff-9a15-d54f80e27119","Type":"ContainerDied","Data":"05be33aa4549a1e9d07bb6fa518319b3503c7b9969c388f0d2c822f529f0294c"}
Nov 03 19:55:01.254030 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:01.253985    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz"
Nov 03 19:55:01.254672 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:01.254041    5591 scope.go:122] "RemoveContainer" containerID="da510dab80149c4e4d1dbcdffb34043327a995c8d630baa06a334d91e1a565e2"
Nov 03 19:55:01.279346 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:01.279312    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz"]
Nov 03 19:55:01.288243 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:01.287322    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-cc7720-7464f494bc-rjzsz"]
Nov 03 19:55:01.350008 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:01.349975    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="727282fc-9c33-4102-96c1-de9b84f0d5d9" path="/var/lib/kubelet/pods/727282fc-9c33-4102-96c1-de9b84f0d5d9/volumes"
Nov 03 19:55:01.350456 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:01.350443    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7510db95-54ae-4fff-9a15-d54f80e27119" path="/var/lib/kubelet/pods/7510db95-54ae-4fff-9a15-d54f80e27119/volumes"
Nov 03 19:55:02.323465 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:02.323429    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"]
Nov 03 19:55:02.323970 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:02.323729    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn" podUID="fc0e2592-7378-4f51-99b4-6b4f06ecc61c" containerName="istio-proxy" containerID="cri-o://570929fa37a1956fdc84f00b4d511dc6b278759febce6527110fa1eed9a78b9c" gracePeriod=30
Nov 03 19:55:04.312241 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:04.312192    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj"]
Nov 03 19:55:04.326999 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:04.326968    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g"]
Nov 03 19:55:05.183779 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:05.183738    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv"]
Nov 03 19:55:05.197976 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:05.197940    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx"]
Nov 03 19:55:06.148177 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:06.148142    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d"]
Nov 03 19:55:06.148789 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:06.148427    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d" podUID="da8f2c4a-8edd-450c-b456-dc6ed51dc851" containerName="maas-api" containerID="cri-o://89a89d30fd6abf625ae7ec8a4fc7b4169df17aff33d8bb9cd12b4585d4c15813" gracePeriod=30
Nov 03 19:55:06.297702 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:06.297664    5591 generic.go:356] "Generic (PLEG): container finished" podID="da8f2c4a-8edd-450c-b456-dc6ed51dc851" containerID="89a89d30fd6abf625ae7ec8a4fc7b4169df17aff33d8bb9cd12b4585d4c15813" exitCode=0
Nov 03 19:55:06.297888 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:06.297813    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d" event={"ID":"da8f2c4a-8edd-450c-b456-dc6ed51dc851","Type":"ContainerDied","Data":"89a89d30fd6abf625ae7ec8a4fc7b4169df17aff33d8bb9cd12b4585d4c15813"}
Nov 03 19:55:06.696631 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:06.689599    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d"
Nov 03 19:55:06.781498 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:06.781466    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da8f2c4a-8edd-450c-b456-dc6ed51dc851-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da8f2c4a-8edd-450c-b456-dc6ed51dc851-metrics-certs\") pod \"da8f2c4a-8edd-450c-b456-dc6ed51dc851\" (UID: \"da8f2c4a-8edd-450c-b456-dc6ed51dc851\") "
Nov 03 19:55:06.781685 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:06.781641    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da8f2c4a-8edd-450c-b456-dc6ed51dc851-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da8f2c4a-8edd-450c-b456-dc6ed51dc851-maas-api-tls\") pod \"da8f2c4a-8edd-450c-b456-dc6ed51dc851\" (UID: \"da8f2c4a-8edd-450c-b456-dc6ed51dc851\") "
Nov 03 19:55:06.781785 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:06.781767    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da8f2c4a-8edd-450c-b456-dc6ed51dc851-kube-api-access-hbcpk\" (UniqueName: \"kubernetes.io/projected/da8f2c4a-8edd-450c-b456-dc6ed51dc851-kube-api-access-hbcpk\") pod \"da8f2c4a-8edd-450c-b456-dc6ed51dc851\" (UID: \"da8f2c4a-8edd-450c-b456-dc6ed51dc851\") "
Nov 03 19:55:06.781850 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:06.781826    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da8f2c4a-8edd-450c-b456-dc6ed51dc851-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8f2c4a-8edd-450c-b456-dc6ed51dc851-openshift-service-ca\") pod \"da8f2c4a-8edd-450c-b456-dc6ed51dc851\" (UID: \"da8f2c4a-8edd-450c-b456-dc6ed51dc851\") "
Nov 03 19:55:06.782179 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:06.782155    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da8f2c4a-8edd-450c-b456-dc6ed51dc851-openshift-service-ca" pod "da8f2c4a-8edd-450c-b456-dc6ed51dc851" (UID: "da8f2c4a-8edd-450c-b456-dc6ed51dc851"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:55:06.782462 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:06.782447    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da8f2c4a-8edd-450c-b456-dc6ed51dc851-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:06.783970 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:06.783946    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8f2c4a-8edd-450c-b456-dc6ed51dc851-kube-api-access-hbcpk" pod "da8f2c4a-8edd-450c-b456-dc6ed51dc851" (UID: "da8f2c4a-8edd-450c-b456-dc6ed51dc851"). InnerVolumeSpecName "kube-api-access-hbcpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:55:06.784439 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:06.784416    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da8f2c4a-8edd-450c-b456-dc6ed51dc851-maas-api-tls" pod "da8f2c4a-8edd-450c-b456-dc6ed51dc851" (UID: "da8f2c4a-8edd-450c-b456-dc6ed51dc851"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:55:06.784529 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:06.784439    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da8f2c4a-8edd-450c-b456-dc6ed51dc851-metrics-certs" pod "da8f2c4a-8edd-450c-b456-dc6ed51dc851" (UID: "da8f2c4a-8edd-450c-b456-dc6ed51dc851"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:55:06.883883 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:06.883849    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da8f2c4a-8edd-450c-b456-dc6ed51dc851-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:06.883883 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:06.883880    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da8f2c4a-8edd-450c-b456-dc6ed51dc851-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:06.883883 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:06.883891    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbcpk\" (UniqueName: \"kubernetes.io/projected/da8f2c4a-8edd-450c-b456-dc6ed51dc851-kube-api-access-hbcpk\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:07.305934 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.305904    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d"
Nov 03 19:55:07.305934 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.305913    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d" event={"ID":"da8f2c4a-8edd-450c-b456-dc6ed51dc851","Type":"ContainerDied","Data":"74823096ff78e01217c84aece4c3d75d57028541cc0a840505a876f88e75d5e2"}
Nov 03 19:55:07.306476 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.305996    5591 scope.go:122] "RemoveContainer" containerID="89a89d30fd6abf625ae7ec8a4fc7b4169df17aff33d8bb9cd12b4585d4c15813"
Nov 03 19:55:07.331765 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.331737    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d"]
Nov 03 19:55:07.338443 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.338418    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ad865c-58bc569d5d-9hl5d"]
Nov 03 19:55:07.352429 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.352396    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da8f2c4a-8edd-450c-b456-dc6ed51dc851" path="/var/lib/kubelet/pods/da8f2c4a-8edd-450c-b456-dc6ed51dc851/volumes"
Nov 03 19:55:07.646124 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.646099    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:55:07.808548 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.808510    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-workload-socket\") pod \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") "
Nov 03 19:55:07.808756 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.808578    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-envoy\") pod \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") "
Nov 03 19:55:07.808756 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.808612    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-podinfo\") pod \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") "
Nov 03 19:55:07.808756 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.808732    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-workload-certs\") pod \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") "
Nov 03 19:55:07.808928 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.808829    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-token\" (UniqueName: \"kubernetes.io/projected/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-token\") pod \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") "
Nov 03 19:55:07.808928 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.808868    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-workload-socket" pod "fc0e2592-7378-4f51-99b4-6b4f06ecc61c" (UID: "fc0e2592-7378-4f51-99b4-6b4f06ecc61c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:55:07.808928 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.808904    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-data\") pod \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") "
Nov 03 19:55:07.809071 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.808958    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-credential-socket\") pod \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") "
Nov 03 19:55:07.809071 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.809034    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-kube-api-access-8tsv4\" (UniqueName: \"kubernetes.io/projected/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-kube-api-access-8tsv4\") pod \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") "
Nov 03 19:55:07.809071 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.809061    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istiod-ca-cert\") pod \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\" (UID: \"fc0e2592-7378-4f51-99b4-6b4f06ecc61c\") "
Nov 03 19:55:07.809228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.809068    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-workload-certs" pod "fc0e2592-7378-4f51-99b4-6b4f06ecc61c" (UID: "fc0e2592-7378-4f51-99b4-6b4f06ecc61c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:55:07.809287 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.809251    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-data" pod "fc0e2592-7378-4f51-99b4-6b4f06ecc61c" (UID: "fc0e2592-7378-4f51-99b4-6b4f06ecc61c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:55:07.809350 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.809289    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-credential-socket" pod "fc0e2592-7378-4f51-99b4-6b4f06ecc61c" (UID: "fc0e2592-7378-4f51-99b4-6b4f06ecc61c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:55:07.809583 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.809558    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istiod-ca-cert" pod "fc0e2592-7378-4f51-99b4-6b4f06ecc61c" (UID: "fc0e2592-7378-4f51-99b4-6b4f06ecc61c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:55:07.810064 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.809993    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:07.810064 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.810022    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:07.810064 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.810036    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:07.810064 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.810049    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:07.810064 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.810061    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:07.811146 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.811122    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-podinfo" pod "fc0e2592-7378-4f51-99b4-6b4f06ecc61c" (UID: "fc0e2592-7378-4f51-99b4-6b4f06ecc61c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:55:07.811275 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.811123    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-envoy" pod "fc0e2592-7378-4f51-99b4-6b4f06ecc61c" (UID: "fc0e2592-7378-4f51-99b4-6b4f06ecc61c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:55:07.811485 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.811469    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-token" pod "fc0e2592-7378-4f51-99b4-6b4f06ecc61c" (UID: "fc0e2592-7378-4f51-99b4-6b4f06ecc61c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:55:07.811623 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.811611    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-kube-api-access-8tsv4" pod "fc0e2592-7378-4f51-99b4-6b4f06ecc61c" (UID: "fc0e2592-7378-4f51-99b4-6b4f06ecc61c"). InnerVolumeSpecName "kube-api-access-8tsv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:55:07.911446 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.911365    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tsv4\" (UniqueName: \"kubernetes.io/projected/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-kube-api-access-8tsv4\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:07.911446 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.911393    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:07.911446 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.911402    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:07.911446 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:07.911411    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc0e2592-7378-4f51-99b4-6b4f06ecc61c-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:08.315576 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:08.315537    5591 generic.go:356] "Generic (PLEG): container finished" podID="fc0e2592-7378-4f51-99b4-6b4f06ecc61c" containerID="570929fa37a1956fdc84f00b4d511dc6b278759febce6527110fa1eed9a78b9c" exitCode=0
Nov 03 19:55:08.316050 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:08.315592    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"
Nov 03 19:55:08.316050 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:08.315609    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn" event={"ID":"fc0e2592-7378-4f51-99b4-6b4f06ecc61c","Type":"ContainerDied","Data":"570929fa37a1956fdc84f00b4d511dc6b278759febce6527110fa1eed9a78b9c"}
Nov 03 19:55:08.316050 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:08.315657    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn" event={"ID":"fc0e2592-7378-4f51-99b4-6b4f06ecc61c","Type":"ContainerDied","Data":"369882f8d90f9a1411b883bebdd4a9c71cc37dbded8d7cd80be1b3269daf4b27"}
Nov 03 19:55:08.316050 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:08.315679    5591 scope.go:122] "RemoveContainer" containerID="570929fa37a1956fdc84f00b4d511dc6b278759febce6527110fa1eed9a78b9c"
Nov 03 19:55:08.327108 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:08.327088    5591 scope.go:122] "RemoveContainer" containerID="44754f7322be589979ca35b6e1215c8814e1921d6996837dc6c14aead87b490c"
Nov 03 19:55:08.337352 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:08.337291    5591 scope.go:122] "RemoveContainer" containerID="570929fa37a1956fdc84f00b4d511dc6b278759febce6527110fa1eed9a78b9c"
Nov 03 19:55:08.337641 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:55:08.337621    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"570929fa37a1956fdc84f00b4d511dc6b278759febce6527110fa1eed9a78b9c\": container with ID starting with 570929fa37a1956fdc84f00b4d511dc6b278759febce6527110fa1eed9a78b9c not found: ID does not exist" containerID="570929fa37a1956fdc84f00b4d511dc6b278759febce6527110fa1eed9a78b9c"
Nov 03 19:55:08.337721 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:08.337650    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"570929fa37a1956fdc84f00b4d511dc6b278759febce6527110fa1eed9a78b9c"} err="failed to get container status \"570929fa37a1956fdc84f00b4d511dc6b278759febce6527110fa1eed9a78b9c\": rpc error: code = NotFound desc = could not find container \"570929fa37a1956fdc84f00b4d511dc6b278759febce6527110fa1eed9a78b9c\": container with ID starting with 570929fa37a1956fdc84f00b4d511dc6b278759febce6527110fa1eed9a78b9c not found: ID does not exist"
Nov 03 19:55:08.337721 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:08.337667    5591 scope.go:122] "RemoveContainer" containerID="44754f7322be589979ca35b6e1215c8814e1921d6996837dc6c14aead87b490c"
Nov 03 19:55:08.337944 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:55:08.337925    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44754f7322be589979ca35b6e1215c8814e1921d6996837dc6c14aead87b490c\": container with ID starting with 44754f7322be589979ca35b6e1215c8814e1921d6996837dc6c14aead87b490c not found: ID does not exist" containerID="44754f7322be589979ca35b6e1215c8814e1921d6996837dc6c14aead87b490c"
Nov 03 19:55:08.338007 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:08.337951    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44754f7322be589979ca35b6e1215c8814e1921d6996837dc6c14aead87b490c"} err="failed to get container status \"44754f7322be589979ca35b6e1215c8814e1921d6996837dc6c14aead87b490c\": rpc error: code = NotFound desc = could not find container \"44754f7322be589979ca35b6e1215c8814e1921d6996837dc6c14aead87b490c\": container with ID starting with 44754f7322be589979ca35b6e1215c8814e1921d6996837dc6c14aead87b490c not found: ID does not exist"
Nov 03 19:55:08.340979 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:08.340944    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"]
Nov 03 19:55:08.346299 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:08.346275    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-cc7720-openshift-default-564f96dcc7-ps6fn"]
Nov 03 19:55:08.787573 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:08.787538    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:55:09.046814 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:09.046742    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz"]
Nov 03 19:55:09.068480 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:09.068447    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb"]
Nov 03 19:55:09.312717 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:09.312626    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj" podUID="b5a7f19b-214a-4785-b023-9311cffdc24f" containerName="payload-processing" containerID="cri-o://20a19c5b76405e3d40192a31e3322231d293171d9b50907b5867c01c33f67b88" gracePeriod=25
Nov 03 19:55:09.375303 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:09.375259    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g" podUID="268bfdfd-019e-4a89-b66f-921a65dfa5ec" containerName="payload-pre-processing" containerID="cri-o://f9b86501d79c9b9ca9571dc9c07f7d121ab5b567a0dff865bfb1e071225b2c73" gracePeriod=25
Nov 03 19:55:09.424608 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:09.420632    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc0e2592-7378-4f51-99b4-6b4f06ecc61c" path="/var/lib/kubelet/pods/fc0e2592-7378-4f51-99b4-6b4f06ecc61c/volumes"
Nov 03 19:55:09.771445 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:09.771416    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj"
Nov 03 19:55:09.820553 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:09.820473    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g"
Nov 03 19:55:09.893172 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:09.893143    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a7f19b-214a-4785-b023-9311cffdc24f-kube-api-access-8nnfb\" (UniqueName: \"kubernetes.io/projected/b5a7f19b-214a-4785-b023-9311cffdc24f-kube-api-access-8nnfb\") pod \"b5a7f19b-214a-4785-b023-9311cffdc24f\" (UID: \"b5a7f19b-214a-4785-b023-9311cffdc24f\") "
Nov 03 19:55:09.893390 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:09.893230    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a7f19b-214a-4785-b023-9311cffdc24f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a7f19b-214a-4785-b023-9311cffdc24f-plugins-config-volume\") pod \"b5a7f19b-214a-4785-b023-9311cffdc24f\" (UID: \"b5a7f19b-214a-4785-b023-9311cffdc24f\") "
Nov 03 19:55:09.893645 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:09.893616    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a7f19b-214a-4785-b023-9311cffdc24f-plugins-config-volume" pod "b5a7f19b-214a-4785-b023-9311cffdc24f" (UID: "b5a7f19b-214a-4785-b023-9311cffdc24f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:55:09.894100 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:09.894079    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a7f19b-214a-4785-b023-9311cffdc24f-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:09.896075 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:09.896047    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a7f19b-214a-4785-b023-9311cffdc24f-kube-api-access-8nnfb" pod "b5a7f19b-214a-4785-b023-9311cffdc24f" (UID: "b5a7f19b-214a-4785-b023-9311cffdc24f"). InnerVolumeSpecName "kube-api-access-8nnfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:55:09.994853 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:09.994822    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/268bfdfd-019e-4a89-b66f-921a65dfa5ec-kube-api-access-c8kp2\" (UniqueName: \"kubernetes.io/projected/268bfdfd-019e-4a89-b66f-921a65dfa5ec-kube-api-access-c8kp2\") pod \"268bfdfd-019e-4a89-b66f-921a65dfa5ec\" (UID: \"268bfdfd-019e-4a89-b66f-921a65dfa5ec\") "
Nov 03 19:55:09.995023 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:09.994937    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/268bfdfd-019e-4a89-b66f-921a65dfa5ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268bfdfd-019e-4a89-b66f-921a65dfa5ec-plugins-config-volume\") pod \"268bfdfd-019e-4a89-b66f-921a65dfa5ec\" (UID: \"268bfdfd-019e-4a89-b66f-921a65dfa5ec\") "
Nov 03 19:55:09.995309 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:09.995282    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/268bfdfd-019e-4a89-b66f-921a65dfa5ec-plugins-config-volume" pod "268bfdfd-019e-4a89-b66f-921a65dfa5ec" (UID: "268bfdfd-019e-4a89-b66f-921a65dfa5ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:55:09.995575 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:09.995559    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nnfb\" (UniqueName: \"kubernetes.io/projected/b5a7f19b-214a-4785-b023-9311cffdc24f-kube-api-access-8nnfb\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:09.995627 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:09.995578    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/268bfdfd-019e-4a89-b66f-921a65dfa5ec-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:09.997633 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:09.997600    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/268bfdfd-019e-4a89-b66f-921a65dfa5ec-kube-api-access-c8kp2" pod "268bfdfd-019e-4a89-b66f-921a65dfa5ec" (UID: "268bfdfd-019e-4a89-b66f-921a65dfa5ec"). InnerVolumeSpecName "kube-api-access-c8kp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:55:10.097196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.097094    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8kp2\" (UniqueName: \"kubernetes.io/projected/268bfdfd-019e-4a89-b66f-921a65dfa5ec-kube-api-access-c8kp2\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:10.193003 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.185326    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv" podUID="37bd5a46-72a3-4db6-b3c6-2e666bb257b5" containerName="payload-processing" containerID="cri-o://cb0c2dce1ffa5d501acb888a340ef0b617d802633c101c34577bf77aeb0eccfa" gracePeriod=25
Nov 03 19:55:10.200529 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.200476    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx" podUID="607277df-ce3e-4124-b752-1256b14e69f4" containerName="payload-pre-processing" containerID="cri-o://69233cf99d220b67859f91f890f094077a8feb900cd7e9f98337874b4af4e958" gracePeriod=25
Nov 03 19:55:10.464493 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.464442    5591 generic.go:356] "Generic (PLEG): container finished" podID="607277df-ce3e-4124-b752-1256b14e69f4" containerID="69233cf99d220b67859f91f890f094077a8feb900cd7e9f98337874b4af4e958" exitCode=0
Nov 03 19:55:10.465040 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.464576    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx" event={"ID":"607277df-ce3e-4124-b752-1256b14e69f4","Type":"ContainerDied","Data":"69233cf99d220b67859f91f890f094077a8feb900cd7e9f98337874b4af4e958"}
Nov 03 19:55:10.475235 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.467100    5591 generic.go:356] "Generic (PLEG): container finished" podID="37bd5a46-72a3-4db6-b3c6-2e666bb257b5" containerID="cb0c2dce1ffa5d501acb888a340ef0b617d802633c101c34577bf77aeb0eccfa" exitCode=0
Nov 03 19:55:10.475235 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.467285    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv" event={"ID":"37bd5a46-72a3-4db6-b3c6-2e666bb257b5","Type":"ContainerDied","Data":"cb0c2dce1ffa5d501acb888a340ef0b617d802633c101c34577bf77aeb0eccfa"}
Nov 03 19:55:10.475235 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.473613    5591 generic.go:356] "Generic (PLEG): container finished" podID="268bfdfd-019e-4a89-b66f-921a65dfa5ec" containerID="f9b86501d79c9b9ca9571dc9c07f7d121ab5b567a0dff865bfb1e071225b2c73" exitCode=0
Nov 03 19:55:10.475235 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.473999    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g"
Nov 03 19:55:10.475235 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.474301    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g" event={"ID":"268bfdfd-019e-4a89-b66f-921a65dfa5ec","Type":"ContainerDied","Data":"f9b86501d79c9b9ca9571dc9c07f7d121ab5b567a0dff865bfb1e071225b2c73"}
Nov 03 19:55:10.475235 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.474381    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g" event={"ID":"268bfdfd-019e-4a89-b66f-921a65dfa5ec","Type":"ContainerDied","Data":"d52c1ef97b6e5056644f16662503227a83041edb37b5735af5403076d877f642"}
Nov 03 19:55:10.475235 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.474423    5591 scope.go:122] "RemoveContainer" containerID="f9b86501d79c9b9ca9571dc9c07f7d121ab5b567a0dff865bfb1e071225b2c73"
Nov 03 19:55:10.489315 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.489282    5591 generic.go:356] "Generic (PLEG): container finished" podID="b5a7f19b-214a-4785-b023-9311cffdc24f" containerID="20a19c5b76405e3d40192a31e3322231d293171d9b50907b5867c01c33f67b88" exitCode=0
Nov 03 19:55:10.489466 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.489343    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj" event={"ID":"b5a7f19b-214a-4785-b023-9311cffdc24f","Type":"ContainerDied","Data":"20a19c5b76405e3d40192a31e3322231d293171d9b50907b5867c01c33f67b88"}
Nov 03 19:55:10.489466 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.489404    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj" event={"ID":"b5a7f19b-214a-4785-b023-9311cffdc24f","Type":"ContainerDied","Data":"5aa2f1e8a7b89c8aad9f46bef0ce994d63a557994b18dc8210da34730e07b2d9"}
Nov 03 19:55:10.489588 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.489508    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj"
Nov 03 19:55:10.523735 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.523712    5591 scope.go:122] "RemoveContainer" containerID="f9b86501d79c9b9ca9571dc9c07f7d121ab5b567a0dff865bfb1e071225b2c73"
Nov 03 19:55:10.524197 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:55:10.524162    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9b86501d79c9b9ca9571dc9c07f7d121ab5b567a0dff865bfb1e071225b2c73\": container with ID starting with f9b86501d79c9b9ca9571dc9c07f7d121ab5b567a0dff865bfb1e071225b2c73 not found: ID does not exist" containerID="f9b86501d79c9b9ca9571dc9c07f7d121ab5b567a0dff865bfb1e071225b2c73"
Nov 03 19:55:10.524329 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.524244    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b86501d79c9b9ca9571dc9c07f7d121ab5b567a0dff865bfb1e071225b2c73"} err="failed to get container status \"f9b86501d79c9b9ca9571dc9c07f7d121ab5b567a0dff865bfb1e071225b2c73\": rpc error: code = NotFound desc = could not find container \"f9b86501d79c9b9ca9571dc9c07f7d121ab5b567a0dff865bfb1e071225b2c73\": container with ID starting with f9b86501d79c9b9ca9571dc9c07f7d121ab5b567a0dff865bfb1e071225b2c73 not found: ID does not exist"
Nov 03 19:55:10.524329 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.524273    5591 scope.go:122] "RemoveContainer" containerID="20a19c5b76405e3d40192a31e3322231d293171d9b50907b5867c01c33f67b88"
Nov 03 19:55:10.557126 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.557101    5591 scope.go:122] "RemoveContainer" containerID="20a19c5b76405e3d40192a31e3322231d293171d9b50907b5867c01c33f67b88"
Nov 03 19:55:10.557520 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:55:10.557492    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20a19c5b76405e3d40192a31e3322231d293171d9b50907b5867c01c33f67b88\": container with ID starting with 20a19c5b76405e3d40192a31e3322231d293171d9b50907b5867c01c33f67b88 not found: ID does not exist" containerID="20a19c5b76405e3d40192a31e3322231d293171d9b50907b5867c01c33f67b88"
Nov 03 19:55:10.557631 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.557532    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20a19c5b76405e3d40192a31e3322231d293171d9b50907b5867c01c33f67b88"} err="failed to get container status \"20a19c5b76405e3d40192a31e3322231d293171d9b50907b5867c01c33f67b88\": rpc error: code = NotFound desc = could not find container \"20a19c5b76405e3d40192a31e3322231d293171d9b50907b5867c01c33f67b88\": container with ID starting with 20a19c5b76405e3d40192a31e3322231d293171d9b50907b5867c01c33f67b88 not found: ID does not exist"
Nov 03 19:55:10.558359 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.558333    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv"
Nov 03 19:55:10.560737 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.560716    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx"
Nov 03 19:55:10.573411 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.573389    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g"]
Nov 03 19:55:10.578750 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.578729    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-dfd6g"]
Nov 03 19:55:10.591568 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.591547    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj"]
Nov 03 19:55:10.595815 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.595794    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-sqjsj"]
Nov 03 19:55:10.707332 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.707245    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/607277df-ce3e-4124-b752-1256b14e69f4-kube-api-access-b6prx\" (UniqueName: \"kubernetes.io/projected/607277df-ce3e-4124-b752-1256b14e69f4-kube-api-access-b6prx\") pod \"607277df-ce3e-4124-b752-1256b14e69f4\" (UID: \"607277df-ce3e-4124-b752-1256b14e69f4\") "
Nov 03 19:55:10.707332 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.707327    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/607277df-ce3e-4124-b752-1256b14e69f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/607277df-ce3e-4124-b752-1256b14e69f4-plugins-config-volume\") pod \"607277df-ce3e-4124-b752-1256b14e69f4\" (UID: \"607277df-ce3e-4124-b752-1256b14e69f4\") "
Nov 03 19:55:10.707575 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.707472    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37bd5a46-72a3-4db6-b3c6-2e666bb257b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37bd5a46-72a3-4db6-b3c6-2e666bb257b5-plugins-config-volume\") pod \"37bd5a46-72a3-4db6-b3c6-2e666bb257b5\" (UID: \"37bd5a46-72a3-4db6-b3c6-2e666bb257b5\") "
Nov 03 19:55:10.707644 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.707592    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37bd5a46-72a3-4db6-b3c6-2e666bb257b5-kube-api-access-fnc96\" (UniqueName: \"kubernetes.io/projected/37bd5a46-72a3-4db6-b3c6-2e666bb257b5-kube-api-access-fnc96\") pod \"37bd5a46-72a3-4db6-b3c6-2e666bb257b5\" (UID: \"37bd5a46-72a3-4db6-b3c6-2e666bb257b5\") "
Nov 03 19:55:10.707832 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.707800    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/607277df-ce3e-4124-b752-1256b14e69f4-plugins-config-volume" pod "607277df-ce3e-4124-b752-1256b14e69f4" (UID: "607277df-ce3e-4124-b752-1256b14e69f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:55:10.708065 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.707944    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37bd5a46-72a3-4db6-b3c6-2e666bb257b5-plugins-config-volume" pod "37bd5a46-72a3-4db6-b3c6-2e666bb257b5" (UID: "37bd5a46-72a3-4db6-b3c6-2e666bb257b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:55:10.708375 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.708349    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/607277df-ce3e-4124-b752-1256b14e69f4-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:10.708441 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.708383    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37bd5a46-72a3-4db6-b3c6-2e666bb257b5-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:10.709655 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.709633    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/607277df-ce3e-4124-b752-1256b14e69f4-kube-api-access-b6prx" pod "607277df-ce3e-4124-b752-1256b14e69f4" (UID: "607277df-ce3e-4124-b752-1256b14e69f4"). InnerVolumeSpecName "kube-api-access-b6prx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:55:10.709797 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.709779    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37bd5a46-72a3-4db6-b3c6-2e666bb257b5-kube-api-access-fnc96" pod "37bd5a46-72a3-4db6-b3c6-2e666bb257b5" (UID: "37bd5a46-72a3-4db6-b3c6-2e666bb257b5"). InnerVolumeSpecName "kube-api-access-fnc96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:55:10.809536 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.809498    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnc96\" (UniqueName: \"kubernetes.io/projected/37bd5a46-72a3-4db6-b3c6-2e666bb257b5-kube-api-access-fnc96\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:10.809536 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:10.809531    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6prx\" (UniqueName: \"kubernetes.io/projected/607277df-ce3e-4124-b752-1256b14e69f4-kube-api-access-b6prx\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:11.352781 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:11.352736    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="268bfdfd-019e-4a89-b66f-921a65dfa5ec" path="/var/lib/kubelet/pods/268bfdfd-019e-4a89-b66f-921a65dfa5ec/volumes"
Nov 03 19:55:11.353228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:11.353213    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5a7f19b-214a-4785-b023-9311cffdc24f" path="/var/lib/kubelet/pods/b5a7f19b-214a-4785-b023-9311cffdc24f/volumes"
Nov 03 19:55:11.498682 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:11.498647    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx"
Nov 03 19:55:11.499239 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:11.498652    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx" event={"ID":"607277df-ce3e-4124-b752-1256b14e69f4","Type":"ContainerDied","Data":"c08e6cdd21f0318644da0b7878941ad0fc7e66ff5d4c2807d8e2fa835bbde0b8"}
Nov 03 19:55:11.499239 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:11.498789    5591 scope.go:122] "RemoveContainer" containerID="69233cf99d220b67859f91f890f094077a8feb900cd7e9f98337874b4af4e958"
Nov 03 19:55:11.500159 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:11.500078    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv"
Nov 03 19:55:11.500159 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:11.500099    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv" event={"ID":"37bd5a46-72a3-4db6-b3c6-2e666bb257b5","Type":"ContainerDied","Data":"ef59c503737a42cda6a7968f9826ba3c25d75bb1ef875b20305bba2501edced0"}
Nov 03 19:55:11.509040 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:11.509025    5591 scope.go:122] "RemoveContainer" containerID="cb0c2dce1ffa5d501acb888a340ef0b617d802633c101c34577bf77aeb0eccfa"
Nov 03 19:55:11.521390 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:11.521359    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv"]
Nov 03 19:55:11.526057 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:11.526032    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ad865c-67b7f75bdb-vwdkv"]
Nov 03 19:55:11.537596 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:11.537572    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx"]
Nov 03 19:55:11.546907 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:11.546880    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ad865c-6f4fbc9955-nbncx"]
Nov 03 19:55:13.353001 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:13.352971    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37bd5a46-72a3-4db6-b3c6-2e666bb257b5" path="/var/lib/kubelet/pods/37bd5a46-72a3-4db6-b3c6-2e666bb257b5/volumes"
Nov 03 19:55:13.353462 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:13.353446    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="607277df-ce3e-4124-b752-1256b14e69f4" path="/var/lib/kubelet/pods/607277df-ce3e-4124-b752-1256b14e69f4/volumes"
Nov 03 19:55:14.047398 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.047352    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz" podUID="aee9f77c-8ccd-4e61-bbb2-5abab68d460b" containerName="payload-processing" containerID="cri-o://693251472c3f25355670145542e426f73810a26a2801198712566f1cb842d303" gracePeriod=25
Nov 03 19:55:14.069501 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.069469    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb" podUID="efa31e2a-d92f-4a8e-98d0-451b78d41880" containerName="payload-pre-processing" containerID="cri-o://a3a417fedf8c60625efdfd660dc319ef3a99f3194f5a4e627e8c79c7a94c6d5f" gracePeriod=25
Nov 03 19:55:14.454769 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.454740    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb"
Nov 03 19:55:14.458281 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.458260    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz"
Nov 03 19:55:14.534886 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.534853    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efa31e2a-d92f-4a8e-98d0-451b78d41880-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efa31e2a-d92f-4a8e-98d0-451b78d41880-plugins-config-volume\") pod \"efa31e2a-d92f-4a8e-98d0-451b78d41880\" (UID: \"efa31e2a-d92f-4a8e-98d0-451b78d41880\") "
Nov 03 19:55:14.535031 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.534893    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efa31e2a-d92f-4a8e-98d0-451b78d41880-kube-api-access-fmbmp\" (UniqueName: \"kubernetes.io/projected/efa31e2a-d92f-4a8e-98d0-451b78d41880-kube-api-access-fmbmp\") pod \"efa31e2a-d92f-4a8e-98d0-451b78d41880\" (UID: \"efa31e2a-d92f-4a8e-98d0-451b78d41880\") "
Nov 03 19:55:14.535031 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.534964    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aee9f77c-8ccd-4e61-bbb2-5abab68d460b-kube-api-access-c84mq\" (UniqueName: \"kubernetes.io/projected/aee9f77c-8ccd-4e61-bbb2-5abab68d460b-kube-api-access-c84mq\") pod \"aee9f77c-8ccd-4e61-bbb2-5abab68d460b\" (UID: \"aee9f77c-8ccd-4e61-bbb2-5abab68d460b\") "
Nov 03 19:55:14.535250 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.535187    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aee9f77c-8ccd-4e61-bbb2-5abab68d460b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aee9f77c-8ccd-4e61-bbb2-5abab68d460b-plugins-config-volume\") pod \"aee9f77c-8ccd-4e61-bbb2-5abab68d460b\" (UID: \"aee9f77c-8ccd-4e61-bbb2-5abab68d460b\") "
Nov 03 19:55:14.535394 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.535256    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efa31e2a-d92f-4a8e-98d0-451b78d41880-plugins-config-volume" pod "efa31e2a-d92f-4a8e-98d0-451b78d41880" (UID: "efa31e2a-d92f-4a8e-98d0-451b78d41880"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:55:14.535573 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.535535    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee9f77c-8ccd-4e61-bbb2-5abab68d460b-plugins-config-volume" pod "aee9f77c-8ccd-4e61-bbb2-5abab68d460b" (UID: "aee9f77c-8ccd-4e61-bbb2-5abab68d460b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:55:14.536309 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.536287    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efa31e2a-d92f-4a8e-98d0-451b78d41880-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:14.536384 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.536315    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aee9f77c-8ccd-4e61-bbb2-5abab68d460b-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:14.537468 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.537447    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee9f77c-8ccd-4e61-bbb2-5abab68d460b-kube-api-access-c84mq" pod "aee9f77c-8ccd-4e61-bbb2-5abab68d460b" (UID: "aee9f77c-8ccd-4e61-bbb2-5abab68d460b"). InnerVolumeSpecName "kube-api-access-c84mq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:55:14.537538 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.537530    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa31e2a-d92f-4a8e-98d0-451b78d41880-kube-api-access-fmbmp" pod "efa31e2a-d92f-4a8e-98d0-451b78d41880" (UID: "efa31e2a-d92f-4a8e-98d0-451b78d41880"). InnerVolumeSpecName "kube-api-access-fmbmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:55:14.564772 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.564735    5591 generic.go:356] "Generic (PLEG): container finished" podID="efa31e2a-d92f-4a8e-98d0-451b78d41880" containerID="a3a417fedf8c60625efdfd660dc319ef3a99f3194f5a4e627e8c79c7a94c6d5f" exitCode=0
Nov 03 19:55:14.564940 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.564785    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb"
Nov 03 19:55:14.564940 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.564814    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb" event={"ID":"efa31e2a-d92f-4a8e-98d0-451b78d41880","Type":"ContainerDied","Data":"a3a417fedf8c60625efdfd660dc319ef3a99f3194f5a4e627e8c79c7a94c6d5f"}
Nov 03 19:55:14.564940 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.564869    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb" event={"ID":"efa31e2a-d92f-4a8e-98d0-451b78d41880","Type":"ContainerDied","Data":"2465a376f413daaf47e009112c2360b8773ab5fec800f8df1c902c422017fd0b"}
Nov 03 19:55:14.564940 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.564891    5591 scope.go:122] "RemoveContainer" containerID="a3a417fedf8c60625efdfd660dc319ef3a99f3194f5a4e627e8c79c7a94c6d5f"
Nov 03 19:55:14.566367 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.566345    5591 generic.go:356] "Generic (PLEG): container finished" podID="aee9f77c-8ccd-4e61-bbb2-5abab68d460b" containerID="693251472c3f25355670145542e426f73810a26a2801198712566f1cb842d303" exitCode=0
Nov 03 19:55:14.566464 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.566408    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz"
Nov 03 19:55:14.566522 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.566468    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz" event={"ID":"aee9f77c-8ccd-4e61-bbb2-5abab68d460b","Type":"ContainerDied","Data":"693251472c3f25355670145542e426f73810a26a2801198712566f1cb842d303"}
Nov 03 19:55:14.566522 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.566509    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz" event={"ID":"aee9f77c-8ccd-4e61-bbb2-5abab68d460b","Type":"ContainerDied","Data":"e4c8518db89433e0cfe54a6f82ffd6d173689fc746a4df05011010d41207de1a"}
Nov 03 19:55:14.577133 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.577097    5591 scope.go:122] "RemoveContainer" containerID="a3a417fedf8c60625efdfd660dc319ef3a99f3194f5a4e627e8c79c7a94c6d5f"
Nov 03 19:55:14.577473 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:55:14.577434    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3a417fedf8c60625efdfd660dc319ef3a99f3194f5a4e627e8c79c7a94c6d5f\": container with ID starting with a3a417fedf8c60625efdfd660dc319ef3a99f3194f5a4e627e8c79c7a94c6d5f not found: ID does not exist" containerID="a3a417fedf8c60625efdfd660dc319ef3a99f3194f5a4e627e8c79c7a94c6d5f"
Nov 03 19:55:14.577567 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.577476    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3a417fedf8c60625efdfd660dc319ef3a99f3194f5a4e627e8c79c7a94c6d5f"} err="failed to get container status \"a3a417fedf8c60625efdfd660dc319ef3a99f3194f5a4e627e8c79c7a94c6d5f\": rpc error: code = NotFound desc = could not find container \"a3a417fedf8c60625efdfd660dc319ef3a99f3194f5a4e627e8c79c7a94c6d5f\": container with ID starting with a3a417fedf8c60625efdfd660dc319ef3a99f3194f5a4e627e8c79c7a94c6d5f not found: ID does not exist"
Nov 03 19:55:14.577567 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.577501    5591 scope.go:122] "RemoveContainer" containerID="693251472c3f25355670145542e426f73810a26a2801198712566f1cb842d303"
Nov 03 19:55:14.588617 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.588595    5591 scope.go:122] "RemoveContainer" containerID="693251472c3f25355670145542e426f73810a26a2801198712566f1cb842d303"
Nov 03 19:55:14.589025 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:55:14.588994    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"693251472c3f25355670145542e426f73810a26a2801198712566f1cb842d303\": container with ID starting with 693251472c3f25355670145542e426f73810a26a2801198712566f1cb842d303 not found: ID does not exist" containerID="693251472c3f25355670145542e426f73810a26a2801198712566f1cb842d303"
Nov 03 19:55:14.589132 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.589036    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"693251472c3f25355670145542e426f73810a26a2801198712566f1cb842d303"} err="failed to get container status \"693251472c3f25355670145542e426f73810a26a2801198712566f1cb842d303\": rpc error: code = NotFound desc = could not find container \"693251472c3f25355670145542e426f73810a26a2801198712566f1cb842d303\": container with ID starting with 693251472c3f25355670145542e426f73810a26a2801198712566f1cb842d303 not found: ID does not exist"
Nov 03 19:55:14.591544 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.591526    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb"]
Nov 03 19:55:14.597758 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.597736    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f20fc0-89489b479-5h6bb"]
Nov 03 19:55:14.608319 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.608187    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz"]
Nov 03 19:55:14.611085 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.611067    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f20fc0-5cdb8d9966-ks9fz"]
Nov 03 19:55:14.637971 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.637940    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmbmp\" (UniqueName: \"kubernetes.io/projected/efa31e2a-d92f-4a8e-98d0-451b78d41880-kube-api-access-fmbmp\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:14.637971 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.637968    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c84mq\" (UniqueName: \"kubernetes.io/projected/aee9f77c-8ccd-4e61-bbb2-5abab68d460b-kube-api-access-c84mq\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:14.727139 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.727104    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78"]
Nov 03 19:55:14.727414 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:14.727370    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78" podUID="af7d8f35-2a5e-4998-a2c9-7637747b374a" containerName="maas-api" containerID="cri-o://d8c94cf025d7b83d5e45ca3acee5ad3ee56f8ff0a710696fe8c3f42253c31685" gracePeriod=30
Nov 03 19:55:15.352886 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:15.352858    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aee9f77c-8ccd-4e61-bbb2-5abab68d460b" path="/var/lib/kubelet/pods/aee9f77c-8ccd-4e61-bbb2-5abab68d460b/volumes"
Nov 03 19:55:15.353350 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:15.353333    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efa31e2a-d92f-4a8e-98d0-451b78d41880" path="/var/lib/kubelet/pods/efa31e2a-d92f-4a8e-98d0-451b78d41880/volumes"
Nov 03 19:55:15.361700 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:15.361680    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78"
Nov 03 19:55:15.542352 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:15.542318    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af7d8f35-2a5e-4998-a2c9-7637747b374a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af7d8f35-2a5e-4998-a2c9-7637747b374a-openshift-service-ca\") pod \"af7d8f35-2a5e-4998-a2c9-7637747b374a\" (UID: \"af7d8f35-2a5e-4998-a2c9-7637747b374a\") "
Nov 03 19:55:15.542923 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:15.542561    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af7d8f35-2a5e-4998-a2c9-7637747b374a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af7d8f35-2a5e-4998-a2c9-7637747b374a-maas-api-tls\") pod \"af7d8f35-2a5e-4998-a2c9-7637747b374a\" (UID: \"af7d8f35-2a5e-4998-a2c9-7637747b374a\") "
Nov 03 19:55:15.542923 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:15.542598    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af7d8f35-2a5e-4998-a2c9-7637747b374a-kube-api-access-lvmr4\" (UniqueName: \"kubernetes.io/projected/af7d8f35-2a5e-4998-a2c9-7637747b374a-kube-api-access-lvmr4\") pod \"af7d8f35-2a5e-4998-a2c9-7637747b374a\" (UID: \"af7d8f35-2a5e-4998-a2c9-7637747b374a\") "
Nov 03 19:55:15.542923 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:15.542614    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af7d8f35-2a5e-4998-a2c9-7637747b374a-openshift-service-ca" pod "af7d8f35-2a5e-4998-a2c9-7637747b374a" (UID: "af7d8f35-2a5e-4998-a2c9-7637747b374a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:55:15.542923 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:15.542670    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af7d8f35-2a5e-4998-a2c9-7637747b374a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/af7d8f35-2a5e-4998-a2c9-7637747b374a-metrics-certs\") pod \"af7d8f35-2a5e-4998-a2c9-7637747b374a\" (UID: \"af7d8f35-2a5e-4998-a2c9-7637747b374a\") "
Nov 03 19:55:15.543479 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:15.543445    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af7d8f35-2a5e-4998-a2c9-7637747b374a-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:15.545187 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:15.545159    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af7d8f35-2a5e-4998-a2c9-7637747b374a-maas-api-tls" pod "af7d8f35-2a5e-4998-a2c9-7637747b374a" (UID: "af7d8f35-2a5e-4998-a2c9-7637747b374a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:55:15.545392 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:15.545190    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af7d8f35-2a5e-4998-a2c9-7637747b374a-kube-api-access-lvmr4" pod "af7d8f35-2a5e-4998-a2c9-7637747b374a" (UID: "af7d8f35-2a5e-4998-a2c9-7637747b374a"). InnerVolumeSpecName "kube-api-access-lvmr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:55:15.545392 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:15.545192    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af7d8f35-2a5e-4998-a2c9-7637747b374a-metrics-certs" pod "af7d8f35-2a5e-4998-a2c9-7637747b374a" (UID: "af7d8f35-2a5e-4998-a2c9-7637747b374a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:55:15.578158 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:15.578079    5591 generic.go:356] "Generic (PLEG): container finished" podID="af7d8f35-2a5e-4998-a2c9-7637747b374a" containerID="d8c94cf025d7b83d5e45ca3acee5ad3ee56f8ff0a710696fe8c3f42253c31685" exitCode=0
Nov 03 19:55:15.578158 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:15.578134    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78"
Nov 03 19:55:15.578420 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:15.578174    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78" event={"ID":"af7d8f35-2a5e-4998-a2c9-7637747b374a","Type":"ContainerDied","Data":"d8c94cf025d7b83d5e45ca3acee5ad3ee56f8ff0a710696fe8c3f42253c31685"}
Nov 03 19:55:15.578420 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:15.578267    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78" event={"ID":"af7d8f35-2a5e-4998-a2c9-7637747b374a","Type":"ContainerDied","Data":"855563b54f0172beefa0b8e0e2a440ec296b4029fb376d51cfa8f93c59992540"}
Nov 03 19:55:15.578420 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:15.578300    5591 scope.go:122] "RemoveContainer" containerID="d8c94cf025d7b83d5e45ca3acee5ad3ee56f8ff0a710696fe8c3f42253c31685"
Nov 03 19:55:15.591798 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:15.591712    5591 scope.go:122] "RemoveContainer" containerID="d8c94cf025d7b83d5e45ca3acee5ad3ee56f8ff0a710696fe8c3f42253c31685"
Nov 03 19:55:15.592144 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:55:15.592124    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8c94cf025d7b83d5e45ca3acee5ad3ee56f8ff0a710696fe8c3f42253c31685\": container with ID starting with d8c94cf025d7b83d5e45ca3acee5ad3ee56f8ff0a710696fe8c3f42253c31685 not found: ID does not exist" containerID="d8c94cf025d7b83d5e45ca3acee5ad3ee56f8ff0a710696fe8c3f42253c31685"
Nov 03 19:55:15.592237 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:15.592151    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c94cf025d7b83d5e45ca3acee5ad3ee56f8ff0a710696fe8c3f42253c31685"} err="failed to get container status \"d8c94cf025d7b83d5e45ca3acee5ad3ee56f8ff0a710696fe8c3f42253c31685\": rpc error: code = NotFound desc = could not find container \"d8c94cf025d7b83d5e45ca3acee5ad3ee56f8ff0a710696fe8c3f42253c31685\": container with ID starting with d8c94cf025d7b83d5e45ca3acee5ad3ee56f8ff0a710696fe8c3f42253c31685 not found: ID does not exist"
Nov 03 19:55:15.604872 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:15.604832    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78"]
Nov 03 19:55:15.608061 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:15.608042    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f20fc0-6b898ddc7b-fcv78"]
Nov 03 19:55:15.644583 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:15.644555    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af7d8f35-2a5e-4998-a2c9-7637747b374a-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:15.644583 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:15.644583    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvmr4\" (UniqueName: \"kubernetes.io/projected/af7d8f35-2a5e-4998-a2c9-7637747b374a-kube-api-access-lvmr4\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:15.644802 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:15.644595    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af7d8f35-2a5e-4998-a2c9-7637747b374a-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:55:17.352069 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:17.352037    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af7d8f35-2a5e-4998-a2c9-7637747b374a" path="/var/lib/kubelet/pods/af7d8f35-2a5e-4998-a2c9-7637747b374a/volumes"
Nov 03 19:55:26.420572 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.420534    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"]
Nov 03 19:55:26.421543 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.421519    5591 cpu_manager.go:414] "containerMap: removing container" podUID="268bfdfd-019e-4a89-b66f-921a65dfa5ec" containerName="payload-pre-processing"
Nov 03 19:55:26.421543 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.421543    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="268bfdfd-019e-4a89-b66f-921a65dfa5ec" containerName="payload-pre-processing"
Nov 03 19:55:26.421713 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.421562    5591 cpu_manager.go:414] "containerMap: removing container" podUID="74f52e2f-1d91-4822-8b8f-05975029d9c2" containerName="maas-api"
Nov 03 19:55:26.421713 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.421570    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f52e2f-1d91-4822-8b8f-05975029d9c2" containerName="maas-api"
Nov 03 19:55:26.421713 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.421580    5591 cpu_manager.go:414] "containerMap: removing container" podUID="aee9f77c-8ccd-4e61-bbb2-5abab68d460b" containerName="payload-processing"
Nov 03 19:55:26.421713 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.421587    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aee9f77c-8ccd-4e61-bbb2-5abab68d460b" containerName="payload-processing"
Nov 03 19:55:26.421713 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.421618    5591 cpu_manager.go:414] "containerMap: removing container" podUID="607277df-ce3e-4124-b752-1256b14e69f4" containerName="payload-pre-processing"
Nov 03 19:55:26.421713 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.421625    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="607277df-ce3e-4124-b752-1256b14e69f4" containerName="payload-pre-processing"
Nov 03 19:55:26.421713 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.421639    5591 cpu_manager.go:414] "containerMap: removing container" podUID="727282fc-9c33-4102-96c1-de9b84f0d5d9" containerName="payload-processing"
Nov 03 19:55:26.421713 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.421646    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="727282fc-9c33-4102-96c1-de9b84f0d5d9" containerName="payload-processing"
Nov 03 19:55:26.421713 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.421665    5591 cpu_manager.go:414] "containerMap: removing container" podUID="fc0e2592-7378-4f51-99b4-6b4f06ecc61c" containerName="istio-proxy"
Nov 03 19:55:26.421713 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.421673    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc0e2592-7378-4f51-99b4-6b4f06ecc61c" containerName="istio-proxy"
Nov 03 19:55:26.421713 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.421685    5591 cpu_manager.go:414] "containerMap: removing container" podUID="7510db95-54ae-4fff-9a15-d54f80e27119" containerName="payload-pre-processing"
Nov 03 19:55:26.421713 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.421691    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7510db95-54ae-4fff-9a15-d54f80e27119" containerName="payload-pre-processing"
Nov 03 19:55:26.421713 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.421699    5591 cpu_manager.go:414] "containerMap: removing container" podUID="fc0e2592-7378-4f51-99b4-6b4f06ecc61c" containerName="istio-proxy"
Nov 03 19:55:26.421713 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.421705    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc0e2592-7378-4f51-99b4-6b4f06ecc61c" containerName="istio-proxy"
Nov 03 19:55:26.422292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.421720    5591 cpu_manager.go:414] "containerMap: removing container" podUID="da8f2c4a-8edd-450c-b456-dc6ed51dc851" containerName="maas-api"
Nov 03 19:55:26.422292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.421727    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da8f2c4a-8edd-450c-b456-dc6ed51dc851" containerName="maas-api"
Nov 03 19:55:26.422292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.421755    5591 cpu_manager.go:414] "containerMap: removing container" podUID="af7d8f35-2a5e-4998-a2c9-7637747b374a" containerName="maas-api"
Nov 03 19:55:26.422292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.421761    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af7d8f35-2a5e-4998-a2c9-7637747b374a" containerName="maas-api"
Nov 03 19:55:26.422292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.421774    5591 cpu_manager.go:414] "containerMap: removing container" podUID="37bd5a46-72a3-4db6-b3c6-2e666bb257b5" containerName="payload-processing"
Nov 03 19:55:26.422292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.421781    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37bd5a46-72a3-4db6-b3c6-2e666bb257b5" containerName="payload-processing"
Nov 03 19:55:26.422292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.421799    5591 cpu_manager.go:414] "containerMap: removing container" podUID="efa31e2a-d92f-4a8e-98d0-451b78d41880" containerName="payload-pre-processing"
Nov 03 19:55:26.422292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.421807    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efa31e2a-d92f-4a8e-98d0-451b78d41880" containerName="payload-pre-processing"
Nov 03 19:55:26.422292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.421857    5591 cpu_manager.go:414] "containerMap: removing container" podUID="b5a7f19b-214a-4785-b023-9311cffdc24f" containerName="payload-processing"
Nov 03 19:55:26.422292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.421865    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a7f19b-214a-4785-b023-9311cffdc24f" containerName="payload-processing"
Nov 03 19:55:26.422292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.422003    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a7f19b-214a-4785-b023-9311cffdc24f" containerName="payload-processing"
Nov 03 19:55:26.422292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.422015    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="268bfdfd-019e-4a89-b66f-921a65dfa5ec" containerName="payload-pre-processing"
Nov 03 19:55:26.422292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.422025    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f52e2f-1d91-4822-8b8f-05975029d9c2" containerName="maas-api"
Nov 03 19:55:26.422292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.422033    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="aee9f77c-8ccd-4e61-bbb2-5abab68d460b" containerName="payload-processing"
Nov 03 19:55:26.422292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.422046    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="607277df-ce3e-4124-b752-1256b14e69f4" containerName="payload-pre-processing"
Nov 03 19:55:26.422292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.422058    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="727282fc-9c33-4102-96c1-de9b84f0d5d9" containerName="payload-processing"
Nov 03 19:55:26.422292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.422069    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc0e2592-7378-4f51-99b4-6b4f06ecc61c" containerName="istio-proxy"
Nov 03 19:55:26.422292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.422077    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7510db95-54ae-4fff-9a15-d54f80e27119" containerName="payload-pre-processing"
Nov 03 19:55:26.422292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.422084    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc0e2592-7378-4f51-99b4-6b4f06ecc61c" containerName="istio-proxy"
Nov 03 19:55:26.422292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.422095    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="da8f2c4a-8edd-450c-b456-dc6ed51dc851" containerName="maas-api"
Nov 03 19:55:26.422292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.422106    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="af7d8f35-2a5e-4998-a2c9-7637747b374a" containerName="maas-api"
Nov 03 19:55:26.422292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.422114    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="37bd5a46-72a3-4db6-b3c6-2e666bb257b5" containerName="payload-processing"
Nov 03 19:55:26.422292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.422125    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="efa31e2a-d92f-4a8e-98d0-451b78d41880" containerName="payload-pre-processing"
Nov 03 19:55:26.426751 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.426733    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:26.436290 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.433981    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c5b820-openshift-default-dockercfg-59l5s\""
Nov 03 19:55:26.453526 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.453494    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"]
Nov 03 19:55:26.571040 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.553234    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-data\") pod \"e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") " pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:26.571393 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.571358    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-token\") pod \"e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") " pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:26.571614 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.571593    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-podinfo\") pod \"e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") " pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:26.571745 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.571731    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-workload-certs\") pod \"e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") " pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:26.571847 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.571834    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-workload-socket\") pod \"e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") " pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:26.572029 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.572011    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-credential-socket\") pod \"e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") " pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:26.572232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.572194    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x99l5\" (UniqueName: \"kubernetes.io/projected/b4bee720-61a7-4d49-918d-2cec14c41bc0-kube-api-access-x99l5\") pod \"e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") " pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:26.581383 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.581343    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-envoy\") pod \"e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") " pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:26.581523 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.581435    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4bee720-61a7-4d49-918d-2cec14c41bc0-istiod-ca-cert\") pod \"e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") " pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:26.687282 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.687161    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-workload-certs\") pod \"e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") " pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:26.693234 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.687518    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-workload-socket\") pod \"e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") " pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:26.693553 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.693529    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-credential-socket\") pod \"e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") " pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:26.694013 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.693996    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x99l5\" (UniqueName: \"kubernetes.io/projected/b4bee720-61a7-4d49-918d-2cec14c41bc0-kube-api-access-x99l5\") pod \"e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") " pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:26.694146 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.694130    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-envoy\") pod \"e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") " pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:26.694286 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.694273    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4bee720-61a7-4d49-918d-2cec14c41bc0-istiod-ca-cert\") pod \"e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") " pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:26.694585 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.694569    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-data\") pod \"e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") " pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:26.699312 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.699286    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-token\") pod \"e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") " pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:26.722514 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.722482    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-podinfo\") pod \"e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") " pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:26.722851 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.693908    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-credential-socket\") pod \"e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") " pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:26.722952 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.695831    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4bee720-61a7-4d49-918d-2cec14c41bc0-istiod-ca-cert\") pod \"e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") " pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:26.723049 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.691611    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-workload-certs\") pod \"e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") " pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:26.723148 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.691807    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-workload-socket\") pod \"e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") " pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:26.723705 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.723684    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-data\") pod \"e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") " pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:26.743235 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.742679    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-token\") pod \"e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") " pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:26.743420 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.743263    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x99l5\" (UniqueName: \"kubernetes.io/projected/b4bee720-61a7-4d49-918d-2cec14c41bc0-kube-api-access-x99l5\") pod \"e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") " pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:26.774264 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.766984    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-podinfo\") pod \"e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") " pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:26.774264 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:26.767177    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-envoy\") pod \"e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") " pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:27.059695 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.059664    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:27.378911 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.378885    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"]
Nov 03 19:55:27.380449 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:55:27.380417    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4bee720_61a7_4d49_918d_2cec14c41bc0.slice/crio-f8a962c252a8b5c7ff6b213bb72a659cd652f65172bd3a15475700c51a4d6733 WatchSource:0}: Error finding container f8a962c252a8b5c7ff6b213bb72a659cd652f65172bd3a15475700c51a4d6733: Status 404 returned error can't find the container with id f8a962c252a8b5c7ff6b213bb72a659cd652f65172bd3a15475700c51a4d6733
Nov 03 19:55:27.383329 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.383295    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:55:27.383407 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.383365    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:55:27.383407 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.383395    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:55:27.393811 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.393783    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh"]
Nov 03 19:55:27.399908 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.399888    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh"
Nov 03 19:55:27.405601 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.405327    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d8d238bf\""
Nov 03 19:55:27.406951 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.406925    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d8d238bf-dockercfg-w2rxc\""
Nov 03 19:55:27.408861 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.408840    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh"]
Nov 03 19:55:27.418276 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.418242    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc"]
Nov 03 19:55:27.425060 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.425019    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc"
Nov 03 19:55:27.430722 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.430695    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc"]
Nov 03 19:55:27.466020 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.465975    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28r48\" (UniqueName: \"kubernetes.io/projected/fa147a82-270a-4322-aaf2-9f7a8e73b82b-kube-api-access-28r48\") pod \"payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh\" (UID: \"fa147a82-270a-4322-aaf2-9f7a8e73b82b\") " pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh"
Nov 03 19:55:27.466458 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.466154    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76760b0b-7d3b-48a5-8c11-44efcec67ef1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc\" (UID: \"76760b0b-7d3b-48a5-8c11-44efcec67ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc"
Nov 03 19:55:27.466458 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.466182    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcqpf\" (UniqueName: \"kubernetes.io/projected/76760b0b-7d3b-48a5-8c11-44efcec67ef1-kube-api-access-gcqpf\") pod \"payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc\" (UID: \"76760b0b-7d3b-48a5-8c11-44efcec67ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc"
Nov 03 19:55:27.466614 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.466453    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa147a82-270a-4322-aaf2-9f7a8e73b82b-plugins-config-volume\") pod \"payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh\" (UID: \"fa147a82-270a-4322-aaf2-9f7a8e73b82b\") " pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh"
Nov 03 19:55:27.582232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.579160    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa147a82-270a-4322-aaf2-9f7a8e73b82b-plugins-config-volume\") pod \"payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh\" (UID: \"fa147a82-270a-4322-aaf2-9f7a8e73b82b\") " pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh"
Nov 03 19:55:27.582232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.579272    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28r48\" (UniqueName: \"kubernetes.io/projected/fa147a82-270a-4322-aaf2-9f7a8e73b82b-kube-api-access-28r48\") pod \"payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh\" (UID: \"fa147a82-270a-4322-aaf2-9f7a8e73b82b\") " pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh"
Nov 03 19:55:27.582232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.579391    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76760b0b-7d3b-48a5-8c11-44efcec67ef1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc\" (UID: \"76760b0b-7d3b-48a5-8c11-44efcec67ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc"
Nov 03 19:55:27.582232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.579417    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcqpf\" (UniqueName: \"kubernetes.io/projected/76760b0b-7d3b-48a5-8c11-44efcec67ef1-kube-api-access-gcqpf\") pod \"payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc\" (UID: \"76760b0b-7d3b-48a5-8c11-44efcec67ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc"
Nov 03 19:55:27.582232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.580771    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa147a82-270a-4322-aaf2-9f7a8e73b82b-plugins-config-volume\") pod \"payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh\" (UID: \"fa147a82-270a-4322-aaf2-9f7a8e73b82b\") " pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh"
Nov 03 19:55:27.582232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.581752    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76760b0b-7d3b-48a5-8c11-44efcec67ef1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc\" (UID: \"76760b0b-7d3b-48a5-8c11-44efcec67ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc"
Nov 03 19:55:27.636326 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.607058    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28r48\" (UniqueName: \"kubernetes.io/projected/fa147a82-270a-4322-aaf2-9f7a8e73b82b-kube-api-access-28r48\") pod \"payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh\" (UID: \"fa147a82-270a-4322-aaf2-9f7a8e73b82b\") " pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh"
Nov 03 19:55:27.636571 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.631975    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcqpf\" (UniqueName: \"kubernetes.io/projected/76760b0b-7d3b-48a5-8c11-44efcec67ef1-kube-api-access-gcqpf\") pod \"payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc\" (UID: \"76760b0b-7d3b-48a5-8c11-44efcec67ef1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc"
Nov 03 19:55:27.736519 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.736483    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh"
Nov 03 19:55:27.797025 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.796773    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc"
Nov 03 19:55:27.829362 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.829327    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr" event={"ID":"b4bee720-61a7-4d49-918d-2cec14c41bc0","Type":"ContainerStarted","Data":"2bb3323200c3e0065d8378450b6320683108c773e62b9185daabf984923934a8"}
Nov 03 19:55:27.829544 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.829394    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr" event={"ID":"b4bee720-61a7-4d49-918d-2cec14c41bc0","Type":"ContainerStarted","Data":"f8a962c252a8b5c7ff6b213bb72a659cd652f65172bd3a15475700c51a4d6733"}
Nov 03 19:55:27.859842 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:27.859223    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr" podStartSLOduration=1.8591870369999999 podStartE2EDuration="1.859187037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:27.853799611 +0000 UTC m=+1076.626092630" watchObservedRunningTime="2026-11-03 19:55:27.859187037 +0000 UTC m=+1076.631480055"
Nov 03 19:55:28.082741 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:28.081275    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:28.097235 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:28.097157    5591 patch_prober.go:29] interesting pod/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:55:28.097403 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:28.097254    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr" podUID="b4bee720-61a7-4d49-918d-2cec14c41bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 19:55:28.139671 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:28.139545    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh"]
Nov 03 19:55:28.141742 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:55:28.141660    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa147a82_270a_4322_aaf2_9f7a8e73b82b.slice/crio-fa9fd6e076c7b9c089f9508cc46fcc72bea6dc28f7c7cd7e5a82b537ddb6373b WatchSource:0}: Error finding container fa9fd6e076c7b9c089f9508cc46fcc72bea6dc28f7c7cd7e5a82b537ddb6373b: Status 404 returned error can't find the container with id fa9fd6e076c7b9c089f9508cc46fcc72bea6dc28f7c7cd7e5a82b537ddb6373b
Nov 03 19:55:28.331910 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:28.331870    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc"]
Nov 03 19:55:28.336100 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:55:28.336073    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76760b0b_7d3b_48a5_8c11_44efcec67ef1.slice/crio-8b95ccc03ed82791855300a40935591471ddf25e82e96ff16551daee628fdb32 WatchSource:0}: Error finding container 8b95ccc03ed82791855300a40935591471ddf25e82e96ff16551daee628fdb32: Status 404 returned error can't find the container with id 8b95ccc03ed82791855300a40935591471ddf25e82e96ff16551daee628fdb32
Nov 03 19:55:28.726538 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:55:28.726451    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:55:28.838826 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:28.838794    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc" event={"ID":"76760b0b-7d3b-48a5-8c11-44efcec67ef1","Type":"ContainerStarted","Data":"2af9095de57ef271fa128861e068013f2097b51dc2a1ac33da09760734e71340"}
Nov 03 19:55:28.839039 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:28.838852    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc" event={"ID":"76760b0b-7d3b-48a5-8c11-44efcec67ef1","Type":"ContainerStarted","Data":"8b95ccc03ed82791855300a40935591471ddf25e82e96ff16551daee628fdb32"}
Nov 03 19:55:28.839039 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:28.838883    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc"
Nov 03 19:55:28.840698 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:28.840651    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh" event={"ID":"fa147a82-270a-4322-aaf2-9f7a8e73b82b","Type":"ContainerStarted","Data":"abeffea6e83b48ca6c0946967ff99f47ba7cd10cc3f34877e242278b996ee55a"}
Nov 03 19:55:28.840827 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:28.840704    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh" event={"ID":"fa147a82-270a-4322-aaf2-9f7a8e73b82b","Type":"ContainerStarted","Data":"fa9fd6e076c7b9c089f9508cc46fcc72bea6dc28f7c7cd7e5a82b537ddb6373b"}
Nov 03 19:55:28.840902 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:28.840858    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh"
Nov 03 19:55:28.856897 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:28.856849    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc" podStartSLOduration=1.856834728 podStartE2EDuration="1.856834728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:28.856469424 +0000 UTC m=+1077.628762440" watchObservedRunningTime="2026-11-03 19:55:28.856834728 +0000 UTC m=+1077.629127746"
Nov 03 19:55:28.886358 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:28.886312    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh" podStartSLOduration=1.886299143 podStartE2EDuration="1.886299143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:28.88291824 +0000 UTC m=+1077.655211260" watchObservedRunningTime="2026-11-03 19:55:28.886299143 +0000 UTC m=+1077.658592160"
Nov 03 19:55:29.076301 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:29.076272    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:29.909268 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:29.894898    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:29.909268 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:29.900053    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:31.396049 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.396022    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6"]
Nov 03 19:55:31.400159 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.400138    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6"
Nov 03 19:55:31.411247 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.408479    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c5b820\""
Nov 03 19:55:31.411247 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.409717    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6"]
Nov 03 19:55:31.459706 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.459668    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr"]
Nov 03 19:55:31.475725 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.475693    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr"
Nov 03 19:55:31.494540 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.494509    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d8d238bf-serving-cert\""
Nov 03 19:55:31.499387 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.499344    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh"]
Nov 03 19:55:31.509677 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.508925    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr"]
Nov 03 19:55:31.509677 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.509035    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh"
Nov 03 19:55:31.510481 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.510450    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f9077db-522f-45f1-af84-383047748cb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6\" (UID: \"2f9077db-522f-45f1-af84-383047748cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6"
Nov 03 19:55:31.510893 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.510871    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptmg2\" (UniqueName: \"kubernetes.io/projected/2f9077db-522f-45f1-af84-383047748cb3-kube-api-access-ptmg2\") pod \"payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6\" (UID: \"2f9077db-522f-45f1-af84-383047748cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6"
Nov 03 19:55:31.515769 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.515744    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh"]
Nov 03 19:55:31.636252 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.596590    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c5b820-dockercfg-dtrlw\""
Nov 03 19:55:31.636252 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.618532    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld94t\" (UniqueName: \"kubernetes.io/projected/3d797d10-ce29-4e3e-a675-8cfbef31bc1e-kube-api-access-ld94t\") pod \"payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh\" (UID: \"3d797d10-ce29-4e3e-a675-8cfbef31bc1e\") " pod="openshift-ingress/payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh"
Nov 03 19:55:31.636252 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.618603    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptmg2\" (UniqueName: \"kubernetes.io/projected/2f9077db-522f-45f1-af84-383047748cb3-kube-api-access-ptmg2\") pod \"payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6\" (UID: \"2f9077db-522f-45f1-af84-383047748cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6"
Nov 03 19:55:31.636252 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.618729    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f9077db-522f-45f1-af84-383047748cb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6\" (UID: \"2f9077db-522f-45f1-af84-383047748cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6"
Nov 03 19:55:31.636252 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.618771    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-maas-api-tls\") pod \"maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr\" (UID: \"fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr"
Nov 03 19:55:31.636252 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.618799    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d797d10-ce29-4e3e-a675-8cfbef31bc1e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh\" (UID: \"3d797d10-ce29-4e3e-a675-8cfbef31bc1e\") " pod="openshift-ingress/payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh"
Nov 03 19:55:31.636252 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.619034    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-metrics-certs\") pod \"maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr\" (UID: \"fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr"
Nov 03 19:55:31.636252 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.619070    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-openshift-service-ca\") pod \"maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr\" (UID: \"fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr"
Nov 03 19:55:31.636252 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.619092    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgnnn\" (UniqueName: \"kubernetes.io/projected/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-kube-api-access-xgnnn\") pod \"maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr\" (UID: \"fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr"
Nov 03 19:55:31.636252 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.620502    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f9077db-522f-45f1-af84-383047748cb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6\" (UID: \"2f9077db-522f-45f1-af84-383047748cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6"
Nov 03 19:55:31.701313 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.687127    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptmg2\" (UniqueName: \"kubernetes.io/projected/2f9077db-522f-45f1-af84-383047748cb3-kube-api-access-ptmg2\") pod \"payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6\" (UID: \"2f9077db-522f-45f1-af84-383047748cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6"
Nov 03 19:55:31.740256 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.718462    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6"
Nov 03 19:55:31.740256 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.723170    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-metrics-certs\") pod \"maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr\" (UID: \"fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr"
Nov 03 19:55:31.740256 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.723243    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-openshift-service-ca\") pod \"maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr\" (UID: \"fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr"
Nov 03 19:55:31.740256 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.723274    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgnnn\" (UniqueName: \"kubernetes.io/projected/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-kube-api-access-xgnnn\") pod \"maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr\" (UID: \"fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr"
Nov 03 19:55:31.740256 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.723309    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld94t\" (UniqueName: \"kubernetes.io/projected/3d797d10-ce29-4e3e-a675-8cfbef31bc1e-kube-api-access-ld94t\") pod \"payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh\" (UID: \"3d797d10-ce29-4e3e-a675-8cfbef31bc1e\") " pod="openshift-ingress/payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh"
Nov 03 19:55:31.740256 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.723466    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-maas-api-tls\") pod \"maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr\" (UID: \"fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr"
Nov 03 19:55:31.740256 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.723493    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d797d10-ce29-4e3e-a675-8cfbef31bc1e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh\" (UID: \"3d797d10-ce29-4e3e-a675-8cfbef31bc1e\") " pod="openshift-ingress/payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh"
Nov 03 19:55:31.740256 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.724421    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d797d10-ce29-4e3e-a675-8cfbef31bc1e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh\" (UID: \"3d797d10-ce29-4e3e-a675-8cfbef31bc1e\") " pod="openshift-ingress/payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh"
Nov 03 19:55:31.779184 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.754239    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-openshift-service-ca\") pod \"maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr\" (UID: \"fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr"
Nov 03 19:55:31.806106 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.806071    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-maas-api-tls\") pod \"maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr\" (UID: \"fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr"
Nov 03 19:55:31.821232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.808812    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-metrics-certs\") pod \"maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr\" (UID: \"fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr"
Nov 03 19:55:31.840232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.822449    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld94t\" (UniqueName: \"kubernetes.io/projected/3d797d10-ce29-4e3e-a675-8cfbef31bc1e-kube-api-access-ld94t\") pod \"payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh\" (UID: \"3d797d10-ce29-4e3e-a675-8cfbef31bc1e\") " pod="openshift-ingress/payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh"
Nov 03 19:55:31.841070 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.841025    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh"
Nov 03 19:55:31.864354 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:31.864140    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgnnn\" (UniqueName: \"kubernetes.io/projected/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-kube-api-access-xgnnn\") pod \"maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr\" (UID: \"fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr"
Nov 03 19:55:32.166635 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:32.166355    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr"
Nov 03 19:55:32.288433 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:32.281049    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6"]
Nov 03 19:55:32.372598 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:32.372066    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh"]
Nov 03 19:55:32.678448 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:32.678421    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr"]
Nov 03 19:55:32.680509 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:55:32.680480    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdce2d9f_ee57_4bc5_9d4d_9f8f8dda0481.slice/crio-4c58795cf308126c51294350ebb0bf57535f5ea33c6727d51a7aad3bdbff8e7a WatchSource:0}: Error finding container 4c58795cf308126c51294350ebb0bf57535f5ea33c6727d51a7aad3bdbff8e7a: Status 404 returned error can't find the container with id 4c58795cf308126c51294350ebb0bf57535f5ea33c6727d51a7aad3bdbff8e7a
Nov 03 19:55:32.977986 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:32.976699    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh" event={"ID":"3d797d10-ce29-4e3e-a675-8cfbef31bc1e","Type":"ContainerStarted","Data":"c38516704804d3cb49f1bb63b0d474314b235ef7a9d96550540ec9e14a81ad5a"}
Nov 03 19:55:32.977986 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:32.976782    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh" event={"ID":"3d797d10-ce29-4e3e-a675-8cfbef31bc1e","Type":"ContainerStarted","Data":"f8233dc364fba7b21d8102483aced7559c7a6e2a93ce0f287a1bdceb6415a920"}
Nov 03 19:55:32.977986 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:32.977909    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh"
Nov 03 19:55:32.981102 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:32.981074    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6" event={"ID":"2f9077db-522f-45f1-af84-383047748cb3","Type":"ContainerStarted","Data":"e3e8a2efc8d13b9b2c0f9dcdb29a3add10c9924c76c80e01444dad7578af77ce"}
Nov 03 19:55:32.989476 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:32.989436    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6" event={"ID":"2f9077db-522f-45f1-af84-383047748cb3","Type":"ContainerStarted","Data":"35733c95f53869c7b19b486e664f5ff215ddcb58081a4d533d761c427807fefb"}
Nov 03 19:55:32.990706 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:32.990684    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6"
Nov 03 19:55:32.993862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:32.993837    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr" event={"ID":"fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481","Type":"ContainerStarted","Data":"4c58795cf308126c51294350ebb0bf57535f5ea33c6727d51a7aad3bdbff8e7a"}
Nov 03 19:55:33.011229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:33.010312    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh" podStartSLOduration=2.010290928 podStartE2EDuration="2.010290928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:33.006774112 +0000 UTC m=+1081.779067123" watchObservedRunningTime="2026-11-03 19:55:33.010290928 +0000 UTC m=+1081.782583946"
Nov 03 19:55:33.035362 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:33.035219    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6" podStartSLOduration=2.035182875 podStartE2EDuration="2.035182875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:33.031683149 +0000 UTC m=+1081.803976167" watchObservedRunningTime="2026-11-03 19:55:33.035182875 +0000 UTC m=+1081.807475974"
Nov 03 19:55:34.002995 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:34.002955    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr" event={"ID":"fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481","Type":"ContainerStarted","Data":"c288a63656a4c99ed4973cb64c2952251a106911daf6b831a4bd4adf45e8c0e3"}
Nov 03 19:55:34.003460 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:34.003051    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr"
Nov 03 19:55:34.024684 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:34.024629    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr" podStartSLOduration=2.455505902 podStartE2EDuration="3.024614841s" totalImagesPullingTime="569.108939ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:34.022858366 +0000 UTC m=+1082.795151385" watchObservedRunningTime="2026-11-03 19:55:34.024614841 +0000 UTC m=+1082.796907858"
Nov 03 19:55:34.836264 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:34.836230    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79"]
Nov 03 19:55:34.842829 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:34.842785    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79"
Nov 03 19:55:34.850764 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:34.850186    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c5b820-serving-cert\""
Nov 03 19:55:34.853394 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:34.853364    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79"]
Nov 03 19:55:34.916108 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:34.916072    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nhvx\" (UniqueName: \"kubernetes.io/projected/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-kube-api-access-5nhvx\") pod \"maas-api-e2e-api-a-c5b820-cbf9886f8-spx79\" (UID: \"4e5befbb-f695-4fc3-9fd3-e9b8191fa235\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79"
Nov 03 19:55:34.916311 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:34.916283    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-maas-api-tls\") pod \"maas-api-e2e-api-a-c5b820-cbf9886f8-spx79\" (UID: \"4e5befbb-f695-4fc3-9fd3-e9b8191fa235\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79"
Nov 03 19:55:34.916374 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:34.916327    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-openshift-service-ca\") pod \"maas-api-e2e-api-a-c5b820-cbf9886f8-spx79\" (UID: \"4e5befbb-f695-4fc3-9fd3-e9b8191fa235\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79"
Nov 03 19:55:34.916374 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:34.916356    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-metrics-certs\") pod \"maas-api-e2e-api-a-c5b820-cbf9886f8-spx79\" (UID: \"4e5befbb-f695-4fc3-9fd3-e9b8191fa235\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79"
Nov 03 19:55:35.035241 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:35.022529    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nhvx\" (UniqueName: \"kubernetes.io/projected/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-kube-api-access-5nhvx\") pod \"maas-api-e2e-api-a-c5b820-cbf9886f8-spx79\" (UID: \"4e5befbb-f695-4fc3-9fd3-e9b8191fa235\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79"
Nov 03 19:55:35.035241 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:55:35.022983    5591 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-serving-cert: secret "maas-api-e2e-api-a-c5b820-serving-cert" not found
Nov 03 19:55:35.035241 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:55:35.023116    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-maas-api-tls podName:4e5befbb-f695-4fc3-9fd3-e9b8191fa235 nodeName:}" failed. No retries permitted until 2026-11-03 19:55:35.52307038 +0000 UTC m=+1084.295363376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-maas-api-tls") pod "maas-api-e2e-api-a-c5b820-cbf9886f8-spx79" (UID: "4e5befbb-f695-4fc3-9fd3-e9b8191fa235") : secret "maas-api-e2e-api-a-c5b820-serving-cert" not found
Nov 03 19:55:35.035241 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:35.022845    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-maas-api-tls\") pod \"maas-api-e2e-api-a-c5b820-cbf9886f8-spx79\" (UID: \"4e5befbb-f695-4fc3-9fd3-e9b8191fa235\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79"
Nov 03 19:55:35.035241 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:35.023519    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-openshift-service-ca\") pod \"maas-api-e2e-api-a-c5b820-cbf9886f8-spx79\" (UID: \"4e5befbb-f695-4fc3-9fd3-e9b8191fa235\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79"
Nov 03 19:55:35.035241 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:35.024432    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-openshift-service-ca\") pod \"maas-api-e2e-api-a-c5b820-cbf9886f8-spx79\" (UID: \"4e5befbb-f695-4fc3-9fd3-e9b8191fa235\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79"
Nov 03 19:55:35.035241 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:35.024547    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-metrics-certs\") pod \"maas-api-e2e-api-a-c5b820-cbf9886f8-spx79\" (UID: \"4e5befbb-f695-4fc3-9fd3-e9b8191fa235\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79"
Nov 03 19:55:35.074231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:35.070976    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nhvx\" (UniqueName: \"kubernetes.io/projected/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-kube-api-access-5nhvx\") pod \"maas-api-e2e-api-a-c5b820-cbf9886f8-spx79\" (UID: \"4e5befbb-f695-4fc3-9fd3-e9b8191fa235\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79"
Nov 03 19:55:35.074231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:35.071535    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-metrics-certs\") pod \"maas-api-e2e-api-a-c5b820-cbf9886f8-spx79\" (UID: \"4e5befbb-f695-4fc3-9fd3-e9b8191fa235\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79"
Nov 03 19:55:35.556231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:35.532919    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-maas-api-tls\") pod \"maas-api-e2e-api-a-c5b820-cbf9886f8-spx79\" (UID: \"4e5befbb-f695-4fc3-9fd3-e9b8191fa235\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79"
Nov 03 19:55:35.565905 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:35.565868    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-maas-api-tls\") pod \"maas-api-e2e-api-a-c5b820-cbf9886f8-spx79\" (UID: \"4e5befbb-f695-4fc3-9fd3-e9b8191fa235\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79"
Nov 03 19:55:35.770940 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:35.770898    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79"
Nov 03 19:55:36.206302 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:36.206221    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79"]
Nov 03 19:55:37.113628 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:37.113596    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79" event={"ID":"4e5befbb-f695-4fc3-9fd3-e9b8191fa235","Type":"ContainerStarted","Data":"c12665f0882a0b91a58165d03cf43fe4f328f604b06bda9aca9c2a18fe764102"}
Nov 03 19:55:37.113832 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:37.113652    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79" event={"ID":"4e5befbb-f695-4fc3-9fd3-e9b8191fa235","Type":"ContainerStarted","Data":"48ab9a7802e4460b9435138278ebcc69a6de29e44e82a4868249d7b0598644b5"}
Nov 03 19:55:37.113832 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:37.113673    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79"
Nov 03 19:55:37.138466 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:37.138413    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79" podStartSLOduration=2.732011406 podStartE2EDuration="3.138397566s" totalImagesPullingTime="406.38616ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:37.134713076 +0000 UTC m=+1085.907006098" watchObservedRunningTime="2026-11-03 19:55:37.138397566 +0000 UTC m=+1085.910690581"
Nov 03 19:55:39.897747 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:39.897713    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh"
Nov 03 19:55:39.898256 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:39.897954    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc"
Nov 03 19:55:40.070524 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:40.070497    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr"
Nov 03 19:55:43.124982 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:43.124901    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79"
Nov 03 19:55:44.673469 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:44.673437    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:55:45.111142 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:45.111108    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6"
Nov 03 19:55:45.112737 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:45.112716    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh"
Nov 03 19:55:46.122344 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:46.122306    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6"]
Nov 03 19:55:46.127327 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:46.127301    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6"
Nov 03 19:55:46.135317 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:46.135295    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-389e6d\""
Nov 03 19:55:46.145341 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:46.145311    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6"]
Nov 03 19:55:46.158127 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:46.158090    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9"]
Nov 03 19:55:46.163958 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:46.163932    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9"
Nov 03 19:55:46.174140 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:46.174112    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-389e6d-dockercfg-5snbd\""
Nov 03 19:55:46.217733 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:46.217694    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9"]
Nov 03 19:55:46.241074 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:46.240549    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4m4j\" (UniqueName: \"kubernetes.io/projected/8c526152-ed75-4788-bdf0-2349937c3526-kube-api-access-b4m4j\") pod \"payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6\" (UID: \"8c526152-ed75-4788-bdf0-2349937c3526\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6"
Nov 03 19:55:46.241074 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:46.240685    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt276\" (UniqueName: \"kubernetes.io/projected/a939f744-ed91-45e6-ac8a-5d2a5f437dbd-kube-api-access-nt276\") pod \"payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9\" (UID: \"a939f744-ed91-45e6-ac8a-5d2a5f437dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9"
Nov 03 19:55:46.241074 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:46.240774    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c526152-ed75-4788-bdf0-2349937c3526-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6\" (UID: \"8c526152-ed75-4788-bdf0-2349937c3526\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6"
Nov 03 19:55:46.241074 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:46.240812    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a939f744-ed91-45e6-ac8a-5d2a5f437dbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9\" (UID: \"a939f744-ed91-45e6-ac8a-5d2a5f437dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9"
Nov 03 19:55:46.346400 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:46.346354    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt276\" (UniqueName: \"kubernetes.io/projected/a939f744-ed91-45e6-ac8a-5d2a5f437dbd-kube-api-access-nt276\") pod \"payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9\" (UID: \"a939f744-ed91-45e6-ac8a-5d2a5f437dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9"
Nov 03 19:55:46.346717 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:46.346697    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c526152-ed75-4788-bdf0-2349937c3526-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6\" (UID: \"8c526152-ed75-4788-bdf0-2349937c3526\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6"
Nov 03 19:55:46.346896 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:46.346881    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a939f744-ed91-45e6-ac8a-5d2a5f437dbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9\" (UID: \"a939f744-ed91-45e6-ac8a-5d2a5f437dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9"
Nov 03 19:55:46.347259 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:46.347232    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4m4j\" (UniqueName: \"kubernetes.io/projected/8c526152-ed75-4788-bdf0-2349937c3526-kube-api-access-b4m4j\") pod \"payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6\" (UID: \"8c526152-ed75-4788-bdf0-2349937c3526\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6"
Nov 03 19:55:46.383312 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:46.380153    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a939f744-ed91-45e6-ac8a-5d2a5f437dbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9\" (UID: \"a939f744-ed91-45e6-ac8a-5d2a5f437dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9"
Nov 03 19:55:46.409230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:46.391489    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c526152-ed75-4788-bdf0-2349937c3526-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6\" (UID: \"8c526152-ed75-4788-bdf0-2349937c3526\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6"
Nov 03 19:55:46.461182 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:46.459910    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4m4j\" (UniqueName: \"kubernetes.io/projected/8c526152-ed75-4788-bdf0-2349937c3526-kube-api-access-b4m4j\") pod \"payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6\" (UID: \"8c526152-ed75-4788-bdf0-2349937c3526\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6"
Nov 03 19:55:46.481034 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:46.480998    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt276\" (UniqueName: \"kubernetes.io/projected/a939f744-ed91-45e6-ac8a-5d2a5f437dbd-kube-api-access-nt276\") pod \"payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9\" (UID: \"a939f744-ed91-45e6-ac8a-5d2a5f437dbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9"
Nov 03 19:55:46.500896 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:46.500553    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9"
Nov 03 19:55:46.745116 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:46.744726    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6"
Nov 03 19:55:47.009652 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:47.009620    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9"]
Nov 03 19:55:47.068956 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:47.067602    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6"]
Nov 03 19:55:47.072225 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:55:47.071755    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c526152_ed75_4788_bdf0_2349937c3526.slice/crio-100e14a21429e09470d68f6dd264b7d9450087ab4e5cd08abb714f8a22f67d7c WatchSource:0}: Error finding container 100e14a21429e09470d68f6dd264b7d9450087ab4e5cd08abb714f8a22f67d7c: Status 404 returned error can't find the container with id 100e14a21429e09470d68f6dd264b7d9450087ab4e5cd08abb714f8a22f67d7c
Nov 03 19:55:47.263960 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:47.263924    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9" event={"ID":"a939f744-ed91-45e6-ac8a-5d2a5f437dbd","Type":"ContainerStarted","Data":"32c0f58e5f4e9ea31e3bfaff23bbdc3338aa8f88f20f05d7b7d3514f62e6676e"}
Nov 03 19:55:47.264437 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:47.264011    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9"
Nov 03 19:55:47.264437 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:47.264062    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9" event={"ID":"a939f744-ed91-45e6-ac8a-5d2a5f437dbd","Type":"ContainerStarted","Data":"9fdbc1c55845c3bf89f07f4794ee8e16effc4429eb5d3e92d85267735a41faa0"}
Nov 03 19:55:47.265659 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:47.265634    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6" event={"ID":"8c526152-ed75-4788-bdf0-2349937c3526","Type":"ContainerStarted","Data":"cf308ffc7203e96e25a42caef5250e5cc9048fa0af9998f028376cdfba16d358"}
Nov 03 19:55:47.265797 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:47.265682    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6" event={"ID":"8c526152-ed75-4788-bdf0-2349937c3526","Type":"ContainerStarted","Data":"100e14a21429e09470d68f6dd264b7d9450087ab4e5cd08abb714f8a22f67d7c"}
Nov 03 19:55:47.265797 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:47.265721    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6"
Nov 03 19:55:47.288275 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:47.287019    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9" podStartSLOduration=1.286998461 podStartE2EDuration="1.286998461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:47.28235597 +0000 UTC m=+1096.054648989" watchObservedRunningTime="2026-11-03 19:55:47.286998461 +0000 UTC m=+1096.059291491"
Nov 03 19:55:47.310955 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:47.310885    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6" podStartSLOduration=1.310865181 podStartE2EDuration="1.310865181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:47.305594498 +0000 UTC m=+1096.077887517" watchObservedRunningTime="2026-11-03 19:55:47.310865181 +0000 UTC m=+1096.083158199"
Nov 03 19:55:47.874292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:47.874251    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb"]
Nov 03 19:55:47.907415 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:47.907382    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb"
Nov 03 19:55:47.914573 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:47.913516    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6cd7f4\""
Nov 03 19:55:47.926380 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:47.919512    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6cd7f4-dockercfg-59vs2\""
Nov 03 19:55:47.934378 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:47.930435    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08ebfbb6-454c-4aba-8c39-a64ec8878154-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb\" (UID: \"08ebfbb6-454c-4aba-8c39-a64ec8878154\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb"
Nov 03 19:55:47.934378 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:47.930563    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gr6t\" (UniqueName: \"kubernetes.io/projected/08ebfbb6-454c-4aba-8c39-a64ec8878154-kube-api-access-6gr6t\") pod \"payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb\" (UID: \"08ebfbb6-454c-4aba-8c39-a64ec8878154\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb"
Nov 03 19:55:47.934378 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:47.932738    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb"]
Nov 03 19:55:47.945297 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:47.945196    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd"]
Nov 03 19:55:47.952622 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:47.951148    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd"
Nov 03 19:55:47.958174 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:47.957948    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd"]
Nov 03 19:55:48.075230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:48.065932    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08ebfbb6-454c-4aba-8c39-a64ec8878154-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb\" (UID: \"08ebfbb6-454c-4aba-8c39-a64ec8878154\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb"
Nov 03 19:55:48.075230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:48.066027    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8bsg\" (UniqueName: \"kubernetes.io/projected/f8a22a57-ff2f-407d-8a50-90c8a26ec6ff-kube-api-access-g8bsg\") pod \"payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd\" (UID: \"f8a22a57-ff2f-407d-8a50-90c8a26ec6ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd"
Nov 03 19:55:48.077380 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:48.075556    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gr6t\" (UniqueName: \"kubernetes.io/projected/08ebfbb6-454c-4aba-8c39-a64ec8878154-kube-api-access-6gr6t\") pod \"payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb\" (UID: \"08ebfbb6-454c-4aba-8c39-a64ec8878154\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb"
Nov 03 19:55:48.077380 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:48.075666    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a22a57-ff2f-407d-8a50-90c8a26ec6ff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd\" (UID: \"f8a22a57-ff2f-407d-8a50-90c8a26ec6ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd"
Nov 03 19:55:48.077380 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:48.076971    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08ebfbb6-454c-4aba-8c39-a64ec8878154-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb\" (UID: \"08ebfbb6-454c-4aba-8c39-a64ec8878154\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb"
Nov 03 19:55:48.238454 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:48.238359    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a22a57-ff2f-407d-8a50-90c8a26ec6ff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd\" (UID: \"f8a22a57-ff2f-407d-8a50-90c8a26ec6ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd"
Nov 03 19:55:48.258358 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:48.241437    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a22a57-ff2f-407d-8a50-90c8a26ec6ff-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd\" (UID: \"f8a22a57-ff2f-407d-8a50-90c8a26ec6ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd"
Nov 03 19:55:48.272870 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:48.272830    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8bsg\" (UniqueName: \"kubernetes.io/projected/f8a22a57-ff2f-407d-8a50-90c8a26ec6ff-kube-api-access-g8bsg\") pod \"payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd\" (UID: \"f8a22a57-ff2f-407d-8a50-90c8a26ec6ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd"
Nov 03 19:55:48.319235 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:48.313894    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gr6t\" (UniqueName: \"kubernetes.io/projected/08ebfbb6-454c-4aba-8c39-a64ec8878154-kube-api-access-6gr6t\") pod \"payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb\" (UID: \"08ebfbb6-454c-4aba-8c39-a64ec8878154\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb"
Nov 03 19:55:48.460105 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:48.448724    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8bsg\" (UniqueName: \"kubernetes.io/projected/f8a22a57-ff2f-407d-8a50-90c8a26ec6ff-kube-api-access-g8bsg\") pod \"payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd\" (UID: \"f8a22a57-ff2f-407d-8a50-90c8a26ec6ff\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd"
Nov 03 19:55:48.540472 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:48.540441    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb"
Nov 03 19:55:48.605278 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:48.603022    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd"
Nov 03 19:55:48.855742 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:48.855703    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb"]
Nov 03 19:55:48.856564 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:55:48.856539    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ebfbb6_454c_4aba_8c39_a64ec8878154.slice/crio-159c283248c4ac4195d24c70168c8cc0f1b60b692183ea69a87a2a37baa3fa83 WatchSource:0}: Error finding container 159c283248c4ac4195d24c70168c8cc0f1b60b692183ea69a87a2a37baa3fa83: Status 404 returned error can't find the container with id 159c283248c4ac4195d24c70168c8cc0f1b60b692183ea69a87a2a37baa3fa83
Nov 03 19:55:48.940217 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:48.940180    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd"]
Nov 03 19:55:48.941592 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:55:48.941564    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a22a57_ff2f_407d_8a50_90c8a26ec6ff.slice/crio-04389bf363ef837cbb8435d34e22014f131f3d824d1218f8f8aeecfdf2e01ff3 WatchSource:0}: Error finding container 04389bf363ef837cbb8435d34e22014f131f3d824d1218f8f8aeecfdf2e01ff3: Status 404 returned error can't find the container with id 04389bf363ef837cbb8435d34e22014f131f3d824d1218f8f8aeecfdf2e01ff3
Nov 03 19:55:49.467522 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:49.467401    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd" event={"ID":"f8a22a57-ff2f-407d-8a50-90c8a26ec6ff","Type":"ContainerStarted","Data":"47d26a33e694ab043cd22b4c5f074085d9b11a65a14677b902db0ff391552516"}
Nov 03 19:55:49.467522 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:49.467469    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd"
Nov 03 19:55:49.467522 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:49.467494    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd" event={"ID":"f8a22a57-ff2f-407d-8a50-90c8a26ec6ff","Type":"ContainerStarted","Data":"04389bf363ef837cbb8435d34e22014f131f3d824d1218f8f8aeecfdf2e01ff3"}
Nov 03 19:55:49.469167 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:49.469142    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb" event={"ID":"08ebfbb6-454c-4aba-8c39-a64ec8878154","Type":"ContainerStarted","Data":"a7c0744b422e3d56028fd5b77c1f94f4afad495f240f99e49d23d678a4b238da"}
Nov 03 19:55:49.469295 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:49.469193    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb"
Nov 03 19:55:49.469295 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:49.469241    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb" event={"ID":"08ebfbb6-454c-4aba-8c39-a64ec8878154","Type":"ContainerStarted","Data":"159c283248c4ac4195d24c70168c8cc0f1b60b692183ea69a87a2a37baa3fa83"}
Nov 03 19:55:49.520951 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:49.520896    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd" podStartSLOduration=2.520880413 podStartE2EDuration="2.520880413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:49.501477402 +0000 UTC m=+1098.273770405" watchObservedRunningTime="2026-11-03 19:55:49.520880413 +0000 UTC m=+1098.293173452"
Nov 03 19:55:49.523864 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:49.523825    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb" podStartSLOduration=2.523812971 podStartE2EDuration="2.523812971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:49.520062915 +0000 UTC m=+1098.292355935" watchObservedRunningTime="2026-11-03 19:55:49.523812971 +0000 UTC m=+1098.296105988"
Nov 03 19:55:53.571583 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:53.571515    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5"]
Nov 03 19:55:53.581420 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:53.580549    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5"
Nov 03 19:55:53.582416 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:53.582399    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5"]
Nov 03 19:55:53.586229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:53.584582    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-6cd7f4-serving-cert\""
Nov 03 19:55:53.598594 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:53.598569    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27q4c\" (UniqueName: \"kubernetes.io/projected/b6d39933-bb86-4a0a-ab3f-bd7265c500af-kube-api-access-27q4c\") pod \"maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5\" (UID: \"b6d39933-bb86-4a0a-ab3f-bd7265c500af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5"
Nov 03 19:55:53.598776 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:53.598756    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6d39933-bb86-4a0a-ab3f-bd7265c500af-metrics-certs\") pod \"maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5\" (UID: \"b6d39933-bb86-4a0a-ab3f-bd7265c500af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5"
Nov 03 19:55:53.598828 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:53.598790    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d39933-bb86-4a0a-ab3f-bd7265c500af-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5\" (UID: \"b6d39933-bb86-4a0a-ab3f-bd7265c500af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5"
Nov 03 19:55:53.598867 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:53.598850    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6d39933-bb86-4a0a-ab3f-bd7265c500af-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5\" (UID: \"b6d39933-bb86-4a0a-ab3f-bd7265c500af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5"
Nov 03 19:55:53.700597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:53.700561    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6d39933-bb86-4a0a-ab3f-bd7265c500af-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5\" (UID: \"b6d39933-bb86-4a0a-ab3f-bd7265c500af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5"
Nov 03 19:55:53.700822 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:53.700696    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27q4c\" (UniqueName: \"kubernetes.io/projected/b6d39933-bb86-4a0a-ab3f-bd7265c500af-kube-api-access-27q4c\") pod \"maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5\" (UID: \"b6d39933-bb86-4a0a-ab3f-bd7265c500af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5"
Nov 03 19:55:53.700904 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:53.700850    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6d39933-bb86-4a0a-ab3f-bd7265c500af-metrics-certs\") pod \"maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5\" (UID: \"b6d39933-bb86-4a0a-ab3f-bd7265c500af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5"
Nov 03 19:55:53.700904 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:53.700870    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d39933-bb86-4a0a-ab3f-bd7265c500af-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5\" (UID: \"b6d39933-bb86-4a0a-ab3f-bd7265c500af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5"
Nov 03 19:55:53.701610 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:53.701582    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d39933-bb86-4a0a-ab3f-bd7265c500af-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5\" (UID: \"b6d39933-bb86-4a0a-ab3f-bd7265c500af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5"
Nov 03 19:55:53.703395 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:53.703368    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6d39933-bb86-4a0a-ab3f-bd7265c500af-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5\" (UID: \"b6d39933-bb86-4a0a-ab3f-bd7265c500af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5"
Nov 03 19:55:53.703511 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:53.703416    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6d39933-bb86-4a0a-ab3f-bd7265c500af-metrics-certs\") pod \"maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5\" (UID: \"b6d39933-bb86-4a0a-ab3f-bd7265c500af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5"
Nov 03 19:55:53.713133 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:53.713103    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27q4c\" (UniqueName: \"kubernetes.io/projected/b6d39933-bb86-4a0a-ab3f-bd7265c500af-kube-api-access-27q4c\") pod \"maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5\" (UID: \"b6d39933-bb86-4a0a-ab3f-bd7265c500af\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5"
Nov 03 19:55:53.915275 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:53.904477    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5"
Nov 03 19:55:54.450494 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:54.446764    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5"]
Nov 03 19:55:54.460945 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:55:54.459404    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d39933_bb86_4a0a_ab3f_bd7265c500af.slice/crio-f181c872ccb2b9e7a0fea30f20f3009b8a30e9d0d4bda5dbfed5e3afb794a5f0 WatchSource:0}: Error finding container f181c872ccb2b9e7a0fea30f20f3009b8a30e9d0d4bda5dbfed5e3afb794a5f0: Status 404 returned error can't find the container with id f181c872ccb2b9e7a0fea30f20f3009b8a30e9d0d4bda5dbfed5e3afb794a5f0
Nov 03 19:55:54.540662 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:54.540619    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5" event={"ID":"b6d39933-bb86-4a0a-ab3f-bd7265c500af","Type":"ContainerStarted","Data":"f181c872ccb2b9e7a0fea30f20f3009b8a30e9d0d4bda5dbfed5e3afb794a5f0"}
Nov 03 19:55:55.551993 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:55.551939    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5" event={"ID":"b6d39933-bb86-4a0a-ab3f-bd7265c500af","Type":"ContainerStarted","Data":"c1687ec5f8d0756b7b72e86c7a38ec932515d730d5d6e0970a61e9bf037d090e"}
Nov 03 19:55:55.552483 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:55.552047    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5"
Nov 03 19:55:55.576707 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:55.576647    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5" podStartSLOduration=2.026507916 podStartE2EDuration="2.576626324s" totalImagesPullingTime="550.118408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:55:55.572233573 +0000 UTC m=+1104.344526591" watchObservedRunningTime="2026-11-03 19:55:55.576626324 +0000 UTC m=+1104.348919342"
Nov 03 19:55:57.567783 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:57.567696    5591 generic.go:356] "Generic (PLEG): container finished" podID="b4bee720-61a7-4d49-918d-2cec14c41bc0" containerID="2bb3323200c3e0065d8378450b6320683108c773e62b9185daabf984923934a8" exitCode=0
Nov 03 19:55:57.567783 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:57.567750    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr" event={"ID":"b4bee720-61a7-4d49-918d-2cec14c41bc0","Type":"ContainerDied","Data":"2bb3323200c3e0065d8378450b6320683108c773e62b9185daabf984923934a8"}
Nov 03 19:55:57.568319 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:57.568265    5591 scope.go:122] "RemoveContainer" containerID="2bb3323200c3e0065d8378450b6320683108c773e62b9185daabf984923934a8"
Nov 03 19:55:57.569952 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:57.569918    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:55:57.570017 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:57.569993    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:55:57.570062 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:57.570032    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:55:58.461168 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:58.461139    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9"
Nov 03 19:55:58.461374 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:58.461222    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6"
Nov 03 19:55:58.606468 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:58.606419    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr" event={"ID":"b4bee720-61a7-4d49-918d-2cec14c41bc0","Type":"ContainerStarted","Data":"42851fddf9898ce74631413f1755b6ec03027b177f6863f9bafddc4024b1cb12"}
Nov 03 19:55:58.844319 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:55:58.844287    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:55:59.060366 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:59.060334    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:55:59.061439 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:59.061408    5591 patch_prober.go:29] interesting pod/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:55:59.061572 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:59.061460    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr" podUID="b4bee720-61a7-4d49-918d-2cec14c41bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 19:55:59.899232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:55:59.898315    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:56:00.060973 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:00.060931    5591 patch_prober.go:29] interesting pod/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:56:00.061190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:00.061015    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr" podUID="b4bee720-61a7-4d49-918d-2cec14c41bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 19:56:00.503188 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:00.503158    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd"
Nov 03 19:56:00.505150 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:00.505129    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb"
Nov 03 19:56:01.060076 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:01.060041    5591 patch_prober.go:29] interesting pod/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:56:01.060509 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:01.060111    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr" podUID="b4bee720-61a7-4d49-918d-2cec14c41bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 19:56:01.588278 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:01.582076    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5"
Nov 03 19:56:02.060494 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:02.060465    5591 patch_prober.go:29] interesting pod/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:56:02.060910 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:02.060524    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr" podUID="b4bee720-61a7-4d49-918d-2cec14c41bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 19:56:03.060923 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:03.060886    5591 patch_prober.go:29] interesting pod/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:56:03.061464 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:03.060953    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr" podUID="b4bee720-61a7-4d49-918d-2cec14c41bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 19:56:04.060526 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:04.060493    5591 patch_prober.go:29] interesting pod/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:56:04.060711 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:04.060550    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr" podUID="b4bee720-61a7-4d49-918d-2cec14c41bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 19:56:04.704981 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:04.704946    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6"]
Nov 03 19:56:04.716677 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:04.716648    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9"]
Nov 03 19:56:05.224233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:05.219472    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:56:05.229092 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:05.229065    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:56:07.068549 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:07.068516    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:56:09.534648 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:09.534612    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66"]
Nov 03 19:56:09.542100 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:09.542072    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66"
Nov 03 19:56:09.548040 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:09.548012    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b0f9d6-serving-cert\""
Nov 03 19:56:09.552034 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:09.552005    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66"]
Nov 03 19:56:09.705261 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:09.701398    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cca4af3-235a-4f69-8cad-a752fea612f6-maas-api-tls\") pod \"maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66\" (UID: \"7cca4af3-235a-4f69-8cad-a752fea612f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66"
Nov 03 19:56:09.705261 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:09.701544    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cca4af3-235a-4f69-8cad-a752fea612f6-openshift-service-ca\") pod \"maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66\" (UID: \"7cca4af3-235a-4f69-8cad-a752fea612f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66"
Nov 03 19:56:09.705261 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:09.701791    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ztxd\" (UniqueName: \"kubernetes.io/projected/7cca4af3-235a-4f69-8cad-a752fea612f6-kube-api-access-2ztxd\") pod \"maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66\" (UID: \"7cca4af3-235a-4f69-8cad-a752fea612f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66"
Nov 03 19:56:09.705261 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:09.701900    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cca4af3-235a-4f69-8cad-a752fea612f6-metrics-certs\") pod \"maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66\" (UID: \"7cca4af3-235a-4f69-8cad-a752fea612f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66"
Nov 03 19:56:09.714243 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:09.712342    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6" podUID="8c526152-ed75-4788-bdf0-2349937c3526" containerName="payload-processing" containerID="cri-o://cf308ffc7203e96e25a42caef5250e5cc9048fa0af9998f028376cdfba16d358" gracePeriod=25
Nov 03 19:56:09.733796 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:09.733750    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9" podUID="a939f744-ed91-45e6-ac8a-5d2a5f437dbd" containerName="payload-pre-processing" containerID="cri-o://32c0f58e5f4e9ea31e3bfaff23bbdc3338aa8f88f20f05d7b7d3514f62e6676e" gracePeriod=25
Nov 03 19:56:09.804298 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:09.804153    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cca4af3-235a-4f69-8cad-a752fea612f6-openshift-service-ca\") pod \"maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66\" (UID: \"7cca4af3-235a-4f69-8cad-a752fea612f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66"
Nov 03 19:56:09.804500 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:09.804413    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ztxd\" (UniqueName: \"kubernetes.io/projected/7cca4af3-235a-4f69-8cad-a752fea612f6-kube-api-access-2ztxd\") pod \"maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66\" (UID: \"7cca4af3-235a-4f69-8cad-a752fea612f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66"
Nov 03 19:56:09.804500 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:09.804477    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cca4af3-235a-4f69-8cad-a752fea612f6-metrics-certs\") pod \"maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66\" (UID: \"7cca4af3-235a-4f69-8cad-a752fea612f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66"
Nov 03 19:56:09.804631 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:09.804546    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cca4af3-235a-4f69-8cad-a752fea612f6-maas-api-tls\") pod \"maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66\" (UID: \"7cca4af3-235a-4f69-8cad-a752fea612f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66"
Nov 03 19:56:09.804722 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:56:09.804704    5591 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-serving-cert: secret "maas-api-e2e-api-b-b0f9d6-serving-cert" not found
Nov 03 19:56:09.804837 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:56:09.804780    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cca4af3-235a-4f69-8cad-a752fea612f6-maas-api-tls podName:7cca4af3-235a-4f69-8cad-a752fea612f6 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:10.304752036 +0000 UTC m=+1119.077045050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7cca4af3-235a-4f69-8cad-a752fea612f6-maas-api-tls") pod "maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66" (UID: "7cca4af3-235a-4f69-8cad-a752fea612f6") : secret "maas-api-e2e-api-b-b0f9d6-serving-cert" not found
Nov 03 19:56:09.804929 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:09.804911    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cca4af3-235a-4f69-8cad-a752fea612f6-openshift-service-ca\") pod \"maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66\" (UID: \"7cca4af3-235a-4f69-8cad-a752fea612f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66"
Nov 03 19:56:09.807525 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:09.807503    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cca4af3-235a-4f69-8cad-a752fea612f6-metrics-certs\") pod \"maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66\" (UID: \"7cca4af3-235a-4f69-8cad-a752fea612f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66"
Nov 03 19:56:09.837671 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:09.837639    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ztxd\" (UniqueName: \"kubernetes.io/projected/7cca4af3-235a-4f69-8cad-a752fea612f6-kube-api-access-2ztxd\") pod \"maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66\" (UID: \"7cca4af3-235a-4f69-8cad-a752fea612f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66"
Nov 03 19:56:09.901929 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:09.901880    5591 generic.go:356] "Generic (PLEG): container finished" podID="a939f744-ed91-45e6-ac8a-5d2a5f437dbd" containerID="32c0f58e5f4e9ea31e3bfaff23bbdc3338aa8f88f20f05d7b7d3514f62e6676e" exitCode=0
Nov 03 19:56:09.902123 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:09.901952    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9" event={"ID":"a939f744-ed91-45e6-ac8a-5d2a5f437dbd","Type":"ContainerDied","Data":"32c0f58e5f4e9ea31e3bfaff23bbdc3338aa8f88f20f05d7b7d3514f62e6676e"}
Nov 03 19:56:09.903822 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:09.903792    5591 generic.go:356] "Generic (PLEG): container finished" podID="8c526152-ed75-4788-bdf0-2349937c3526" containerID="cf308ffc7203e96e25a42caef5250e5cc9048fa0af9998f028376cdfba16d358" exitCode=0
Nov 03 19:56:09.903979 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:09.903857    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6" event={"ID":"8c526152-ed75-4788-bdf0-2349937c3526","Type":"ContainerDied","Data":"cf308ffc7203e96e25a42caef5250e5cc9048fa0af9998f028376cdfba16d358"}
Nov 03 19:56:09.998232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:09.996500    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd"]
Nov 03 19:56:10.035299 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:10.027901    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb"]
Nov 03 19:56:10.137080 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:10.137054    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6"
Nov 03 19:56:10.139409 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:10.139379    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9"
Nov 03 19:56:10.236679 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:10.236645    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a939f744-ed91-45e6-ac8a-5d2a5f437dbd-kube-api-access-nt276\" (UniqueName: \"kubernetes.io/projected/a939f744-ed91-45e6-ac8a-5d2a5f437dbd-kube-api-access-nt276\") pod \"a939f744-ed91-45e6-ac8a-5d2a5f437dbd\" (UID: \"a939f744-ed91-45e6-ac8a-5d2a5f437dbd\") "
Nov 03 19:56:10.236878 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:10.236706    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c526152-ed75-4788-bdf0-2349937c3526-kube-api-access-b4m4j\" (UniqueName: \"kubernetes.io/projected/8c526152-ed75-4788-bdf0-2349937c3526-kube-api-access-b4m4j\") pod \"8c526152-ed75-4788-bdf0-2349937c3526\" (UID: \"8c526152-ed75-4788-bdf0-2349937c3526\") "
Nov 03 19:56:10.236878 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:10.236780    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c526152-ed75-4788-bdf0-2349937c3526-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c526152-ed75-4788-bdf0-2349937c3526-plugins-config-volume\") pod \"8c526152-ed75-4788-bdf0-2349937c3526\" (UID: \"8c526152-ed75-4788-bdf0-2349937c3526\") "
Nov 03 19:56:10.237035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:10.237013    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a939f744-ed91-45e6-ac8a-5d2a5f437dbd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a939f744-ed91-45e6-ac8a-5d2a5f437dbd-plugins-config-volume\") pod \"a939f744-ed91-45e6-ac8a-5d2a5f437dbd\" (UID: \"a939f744-ed91-45e6-ac8a-5d2a5f437dbd\") "
Nov 03 19:56:10.237271 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:10.237250    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c526152-ed75-4788-bdf0-2349937c3526-plugins-config-volume" pod "8c526152-ed75-4788-bdf0-2349937c3526" (UID: "8c526152-ed75-4788-bdf0-2349937c3526"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:56:10.237353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:10.237332    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a939f744-ed91-45e6-ac8a-5d2a5f437dbd-plugins-config-volume" pod "a939f744-ed91-45e6-ac8a-5d2a5f437dbd" (UID: "a939f744-ed91-45e6-ac8a-5d2a5f437dbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:56:10.237712 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:10.237695    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a939f744-ed91-45e6-ac8a-5d2a5f437dbd-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:10.237755 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:10.237716    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c526152-ed75-4788-bdf0-2349937c3526-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:10.239070 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:10.239052    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a939f744-ed91-45e6-ac8a-5d2a5f437dbd-kube-api-access-nt276" pod "a939f744-ed91-45e6-ac8a-5d2a5f437dbd" (UID: "a939f744-ed91-45e6-ac8a-5d2a5f437dbd"). InnerVolumeSpecName "kube-api-access-nt276". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:56:10.239361 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:10.239346    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c526152-ed75-4788-bdf0-2349937c3526-kube-api-access-b4m4j" pod "8c526152-ed75-4788-bdf0-2349937c3526" (UID: "8c526152-ed75-4788-bdf0-2349937c3526"). InnerVolumeSpecName "kube-api-access-b4m4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:56:10.339314 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:10.339213    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cca4af3-235a-4f69-8cad-a752fea612f6-maas-api-tls\") pod \"maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66\" (UID: \"7cca4af3-235a-4f69-8cad-a752fea612f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66"
Nov 03 19:56:10.339494 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:10.339324    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt276\" (UniqueName: \"kubernetes.io/projected/a939f744-ed91-45e6-ac8a-5d2a5f437dbd-kube-api-access-nt276\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:10.339494 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:10.339338    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4m4j\" (UniqueName: \"kubernetes.io/projected/8c526152-ed75-4788-bdf0-2349937c3526-kube-api-access-b4m4j\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:10.341948 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:10.341918    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cca4af3-235a-4f69-8cad-a752fea612f6-maas-api-tls\") pod \"maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66\" (UID: \"7cca4af3-235a-4f69-8cad-a752fea612f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66"
Nov 03 19:56:10.474232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:10.471617    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66"
Nov 03 19:56:10.943013 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:10.929585    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9" event={"ID":"a939f744-ed91-45e6-ac8a-5d2a5f437dbd","Type":"ContainerDied","Data":"9fdbc1c55845c3bf89f07f4794ee8e16effc4429eb5d3e92d85267735a41faa0"}
Nov 03 19:56:10.943013 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:10.929656    5591 scope.go:122] "RemoveContainer" containerID="32c0f58e5f4e9ea31e3bfaff23bbdc3338aa8f88f20f05d7b7d3514f62e6676e"
Nov 03 19:56:10.943013 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:10.929799    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9"
Nov 03 19:56:11.017231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:10.991115    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6" event={"ID":"8c526152-ed75-4788-bdf0-2349937c3526","Type":"ContainerDied","Data":"100e14a21429e09470d68f6dd264b7d9450087ab4e5cd08abb714f8a22f67d7c"}
Nov 03 19:56:11.017231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:10.991285    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6"
Nov 03 19:56:11.035536 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:11.032574    5591 scope.go:122] "RemoveContainer" containerID="cf308ffc7203e96e25a42caef5250e5cc9048fa0af9998f028376cdfba16d358"
Nov 03 19:56:11.035536 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:11.035515    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9"]
Nov 03 19:56:11.186769 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:11.062188    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-389e6d-76698b8d7-tgdg9"]
Nov 03 19:56:11.212444 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:11.211954    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6"]
Nov 03 19:56:11.217520 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:11.217403    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-389e6d-777d766cf8-cwbm6"]
Nov 03 19:56:11.301059 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:11.301032    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66"]
Nov 03 19:56:11.303330 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:56:11.303301    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cca4af3_235a_4f69_8cad_a752fea612f6.slice/crio-1176eed55abb066d28fe078bac8d5c9571bec58da848f8d204f306fd3b38bafd WatchSource:0}: Error finding container 1176eed55abb066d28fe078bac8d5c9571bec58da848f8d204f306fd3b38bafd: Status 404 returned error can't find the container with id 1176eed55abb066d28fe078bac8d5c9571bec58da848f8d204f306fd3b38bafd
Nov 03 19:56:11.351738 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:11.351702    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c526152-ed75-4788-bdf0-2349937c3526" path="/var/lib/kubelet/pods/8c526152-ed75-4788-bdf0-2349937c3526/volumes"
Nov 03 19:56:11.352287 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:11.352266    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a939f744-ed91-45e6-ac8a-5d2a5f437dbd" path="/var/lib/kubelet/pods/a939f744-ed91-45e6-ac8a-5d2a5f437dbd/volumes"
Nov 03 19:56:11.999809 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:11.999773    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66" event={"ID":"7cca4af3-235a-4f69-8cad-a752fea612f6","Type":"ContainerStarted","Data":"53c82f35ecb9fca90c73db144d85c0e8fe56cd0fe464cc49e511c4ffc77db0f2"}
Nov 03 19:56:12.000303 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:11.999838    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66"
Nov 03 19:56:12.000303 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:11.999862    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66" event={"ID":"7cca4af3-235a-4f69-8cad-a752fea612f6","Type":"ContainerStarted","Data":"1176eed55abb066d28fe078bac8d5c9571bec58da848f8d204f306fd3b38bafd"}
Nov 03 19:56:12.022358 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:12.022306    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66" podStartSLOduration=2.472711619 podStartE2EDuration="3.022288706s" totalImagesPullingTime="549.577087ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:12.018676407 +0000 UTC m=+1120.790969432" watchObservedRunningTime="2026-11-03 19:56:12.022288706 +0000 UTC m=+1120.794581718"
Nov 03 19:56:14.308109 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:14.308071    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb"]
Nov 03 19:56:14.309284 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:14.309260    5591 cpu_manager.go:414] "containerMap: removing container" podUID="a939f744-ed91-45e6-ac8a-5d2a5f437dbd" containerName="payload-pre-processing"
Nov 03 19:56:14.309390 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:14.309286    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a939f744-ed91-45e6-ac8a-5d2a5f437dbd" containerName="payload-pre-processing"
Nov 03 19:56:14.309390 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:14.309328    5591 cpu_manager.go:414] "containerMap: removing container" podUID="8c526152-ed75-4788-bdf0-2349937c3526" containerName="payload-processing"
Nov 03 19:56:14.309390 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:14.309334    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c526152-ed75-4788-bdf0-2349937c3526" containerName="payload-processing"
Nov 03 19:56:14.309487 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:14.309440    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="a939f744-ed91-45e6-ac8a-5d2a5f437dbd" containerName="payload-pre-processing"
Nov 03 19:56:14.309487 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:14.309453    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c526152-ed75-4788-bdf0-2349937c3526" containerName="payload-processing"
Nov 03 19:56:14.313465 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:14.313446    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb"
Nov 03 19:56:14.323883 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:14.323857    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb"]
Nov 03 19:56:14.324633 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:14.324610    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b34f6c\""
Nov 03 19:56:14.519231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:14.515178    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h488f\" (UniqueName: \"kubernetes.io/projected/6a5c139a-00fc-40e3-8917-df4391456425-kube-api-access-h488f\") pod \"payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb\" (UID: \"6a5c139a-00fc-40e3-8917-df4391456425\") " pod="openshift-ingress/payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb"
Nov 03 19:56:14.519231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:14.515470    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5c139a-00fc-40e3-8917-df4391456425-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb\" (UID: \"6a5c139a-00fc-40e3-8917-df4391456425\") " pod="openshift-ingress/payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb"
Nov 03 19:56:14.628590 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:14.628498    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h488f\" (UniqueName: \"kubernetes.io/projected/6a5c139a-00fc-40e3-8917-df4391456425-kube-api-access-h488f\") pod \"payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb\" (UID: \"6a5c139a-00fc-40e3-8917-df4391456425\") " pod="openshift-ingress/payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb"
Nov 03 19:56:14.661927 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:14.661891    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5c139a-00fc-40e3-8917-df4391456425-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb\" (UID: \"6a5c139a-00fc-40e3-8917-df4391456425\") " pod="openshift-ingress/payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb"
Nov 03 19:56:14.679120 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:14.663228    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5c139a-00fc-40e3-8917-df4391456425-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb\" (UID: \"6a5c139a-00fc-40e3-8917-df4391456425\") " pod="openshift-ingress/payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb"
Nov 03 19:56:14.686343 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:14.685616    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h488f\" (UniqueName: \"kubernetes.io/projected/6a5c139a-00fc-40e3-8917-df4391456425-kube-api-access-h488f\") pod \"payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb\" (UID: \"6a5c139a-00fc-40e3-8917-df4391456425\") " pod="openshift-ingress/payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb"
Nov 03 19:56:14.936157 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:14.936069    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb"
Nov 03 19:56:14.997278 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:14.997056    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd" podUID="f8a22a57-ff2f-407d-8a50-90c8a26ec6ff" containerName="payload-processing" containerID="cri-o://47d26a33e694ab043cd22b4c5f074085d9b11a65a14677b902db0ff391552516" gracePeriod=25
Nov 03 19:56:15.030895 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:15.030842    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb" podUID="08ebfbb6-454c-4aba-8c39-a64ec8878154" containerName="payload-pre-processing" containerID="cri-o://a7c0744b422e3d56028fd5b77c1f94f4afad495f240f99e49d23d678a4b238da" gracePeriod=25
Nov 03 19:56:15.583482 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:15.583407    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb"]
Nov 03 19:56:15.643964 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:56:15.643537    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5c139a_00fc_40e3_8917_df4391456425.slice/crio-808d9863346d4b06a2f564d48be72f69ed70117b6253eb3ea2ce32af41062156 WatchSource:0}: Error finding container 808d9863346d4b06a2f564d48be72f69ed70117b6253eb3ea2ce32af41062156: Status 404 returned error can't find the container with id 808d9863346d4b06a2f564d48be72f69ed70117b6253eb3ea2ce32af41062156
Nov 03 19:56:15.810421 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:15.810392    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb"
Nov 03 19:56:15.813994 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:15.813972    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd"
Nov 03 19:56:15.986908 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:15.986570    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8a22a57-ff2f-407d-8a50-90c8a26ec6ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a22a57-ff2f-407d-8a50-90c8a26ec6ff-plugins-config-volume\") pod \"f8a22a57-ff2f-407d-8a50-90c8a26ec6ff\" (UID: \"f8a22a57-ff2f-407d-8a50-90c8a26ec6ff\") "
Nov 03 19:56:15.986908 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:15.986799    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8a22a57-ff2f-407d-8a50-90c8a26ec6ff-kube-api-access-g8bsg\" (UniqueName: \"kubernetes.io/projected/f8a22a57-ff2f-407d-8a50-90c8a26ec6ff-kube-api-access-g8bsg\") pod \"f8a22a57-ff2f-407d-8a50-90c8a26ec6ff\" (UID: \"f8a22a57-ff2f-407d-8a50-90c8a26ec6ff\") "
Nov 03 19:56:15.987173 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:15.986917    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08ebfbb6-454c-4aba-8c39-a64ec8878154-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08ebfbb6-454c-4aba-8c39-a64ec8878154-plugins-config-volume\") pod \"08ebfbb6-454c-4aba-8c39-a64ec8878154\" (UID: \"08ebfbb6-454c-4aba-8c39-a64ec8878154\") "
Nov 03 19:56:15.987173 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:15.986992    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ebfbb6-454c-4aba-8c39-a64ec8878154-kube-api-access-6gr6t\" (UniqueName: \"kubernetes.io/projected/08ebfbb6-454c-4aba-8c39-a64ec8878154-kube-api-access-6gr6t\") pod \"08ebfbb6-454c-4aba-8c39-a64ec8878154\" (UID: \"08ebfbb6-454c-4aba-8c39-a64ec8878154\") "
Nov 03 19:56:15.998383 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:15.996567    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ebfbb6-454c-4aba-8c39-a64ec8878154-kube-api-access-6gr6t" pod "08ebfbb6-454c-4aba-8c39-a64ec8878154" (UID: "08ebfbb6-454c-4aba-8c39-a64ec8878154"). InnerVolumeSpecName "kube-api-access-6gr6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:56:15.998383 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:15.996956    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a22a57-ff2f-407d-8a50-90c8a26ec6ff-plugins-config-volume" pod "f8a22a57-ff2f-407d-8a50-90c8a26ec6ff" (UID: "f8a22a57-ff2f-407d-8a50-90c8a26ec6ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:56:15.998383 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:15.997658    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ebfbb6-454c-4aba-8c39-a64ec8878154-plugins-config-volume" pod "08ebfbb6-454c-4aba-8c39-a64ec8878154" (UID: "08ebfbb6-454c-4aba-8c39-a64ec8878154"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:56:16.002096 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:16.001918    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a22a57-ff2f-407d-8a50-90c8a26ec6ff-kube-api-access-g8bsg" pod "f8a22a57-ff2f-407d-8a50-90c8a26ec6ff" (UID: "f8a22a57-ff2f-407d-8a50-90c8a26ec6ff"). InnerVolumeSpecName "kube-api-access-g8bsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:56:16.089785 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:16.089698    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8a22a57-ff2f-407d-8a50-90c8a26ec6ff-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:16.089785 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:16.089732    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8bsg\" (UniqueName: \"kubernetes.io/projected/f8a22a57-ff2f-407d-8a50-90c8a26ec6ff-kube-api-access-g8bsg\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:16.089785 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:16.089742    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08ebfbb6-454c-4aba-8c39-a64ec8878154-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:16.089785 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:16.089751    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gr6t\" (UniqueName: \"kubernetes.io/projected/08ebfbb6-454c-4aba-8c39-a64ec8878154-kube-api-access-6gr6t\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:16.166853 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:16.166821    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb" event={"ID":"6a5c139a-00fc-40e3-8917-df4391456425","Type":"ContainerStarted","Data":"d647cc15f8b68add527a3a033b5fcf8bbaac01ab3b12ea84ad8e1a24772d1538"}
Nov 03 19:56:16.167052 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:16.166875    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb" event={"ID":"6a5c139a-00fc-40e3-8917-df4391456425","Type":"ContainerStarted","Data":"808d9863346d4b06a2f564d48be72f69ed70117b6253eb3ea2ce32af41062156"}
Nov 03 19:56:16.167052 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:16.166903    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb"
Nov 03 19:56:16.168290 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:16.168267    5591 generic.go:356] "Generic (PLEG): container finished" podID="f8a22a57-ff2f-407d-8a50-90c8a26ec6ff" containerID="47d26a33e694ab043cd22b4c5f074085d9b11a65a14677b902db0ff391552516" exitCode=0
Nov 03 19:56:16.168394 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:16.168303    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd"
Nov 03 19:56:16.168394 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:16.168344    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd" event={"ID":"f8a22a57-ff2f-407d-8a50-90c8a26ec6ff","Type":"ContainerDied","Data":"47d26a33e694ab043cd22b4c5f074085d9b11a65a14677b902db0ff391552516"}
Nov 03 19:56:16.168394 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:16.168390    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd" event={"ID":"f8a22a57-ff2f-407d-8a50-90c8a26ec6ff","Type":"ContainerDied","Data":"04389bf363ef837cbb8435d34e22014f131f3d824d1218f8f8aeecfdf2e01ff3"}
Nov 03 19:56:16.168544 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:16.168413    5591 scope.go:122] "RemoveContainer" containerID="47d26a33e694ab043cd22b4c5f074085d9b11a65a14677b902db0ff391552516"
Nov 03 19:56:16.169847 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:16.169826    5591 generic.go:356] "Generic (PLEG): container finished" podID="08ebfbb6-454c-4aba-8c39-a64ec8878154" containerID="a7c0744b422e3d56028fd5b77c1f94f4afad495f240f99e49d23d678a4b238da" exitCode=0
Nov 03 19:56:16.169949 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:16.169877    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb"
Nov 03 19:56:16.169949 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:16.169909    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb" event={"ID":"08ebfbb6-454c-4aba-8c39-a64ec8878154","Type":"ContainerDied","Data":"a7c0744b422e3d56028fd5b77c1f94f4afad495f240f99e49d23d678a4b238da"}
Nov 03 19:56:16.170038 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:16.169956    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb" event={"ID":"08ebfbb6-454c-4aba-8c39-a64ec8878154","Type":"ContainerDied","Data":"159c283248c4ac4195d24c70168c8cc0f1b60b692183ea69a87a2a37baa3fa83"}
Nov 03 19:56:16.179343 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:16.179296    5591 scope.go:122] "RemoveContainer" containerID="47d26a33e694ab043cd22b4c5f074085d9b11a65a14677b902db0ff391552516"
Nov 03 19:56:16.179662 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:56:16.179635    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47d26a33e694ab043cd22b4c5f074085d9b11a65a14677b902db0ff391552516\": container with ID starting with 47d26a33e694ab043cd22b4c5f074085d9b11a65a14677b902db0ff391552516 not found: ID does not exist" containerID="47d26a33e694ab043cd22b4c5f074085d9b11a65a14677b902db0ff391552516"
Nov 03 19:56:16.179785 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:16.179675    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47d26a33e694ab043cd22b4c5f074085d9b11a65a14677b902db0ff391552516"} err="failed to get container status \"47d26a33e694ab043cd22b4c5f074085d9b11a65a14677b902db0ff391552516\": rpc error: code = NotFound desc = could not find container \"47d26a33e694ab043cd22b4c5f074085d9b11a65a14677b902db0ff391552516\": container with ID starting with 47d26a33e694ab043cd22b4c5f074085d9b11a65a14677b902db0ff391552516 not found: ID does not exist"
Nov 03 19:56:16.179785 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:16.179695    5591 scope.go:122] "RemoveContainer" containerID="a7c0744b422e3d56028fd5b77c1f94f4afad495f240f99e49d23d678a4b238da"
Nov 03 19:56:16.190925 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:16.190899    5591 scope.go:122] "RemoveContainer" containerID="a7c0744b422e3d56028fd5b77c1f94f4afad495f240f99e49d23d678a4b238da"
Nov 03 19:56:16.191335 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:56:16.191311    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7c0744b422e3d56028fd5b77c1f94f4afad495f240f99e49d23d678a4b238da\": container with ID starting with a7c0744b422e3d56028fd5b77c1f94f4afad495f240f99e49d23d678a4b238da not found: ID does not exist" containerID="a7c0744b422e3d56028fd5b77c1f94f4afad495f240f99e49d23d678a4b238da"
Nov 03 19:56:16.191433 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:16.191347    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7c0744b422e3d56028fd5b77c1f94f4afad495f240f99e49d23d678a4b238da"} err="failed to get container status \"a7c0744b422e3d56028fd5b77c1f94f4afad495f240f99e49d23d678a4b238da\": rpc error: code = NotFound desc = could not find container \"a7c0744b422e3d56028fd5b77c1f94f4afad495f240f99e49d23d678a4b238da\": container with ID starting with a7c0744b422e3d56028fd5b77c1f94f4afad495f240f99e49d23d678a4b238da not found: ID does not exist"
Nov 03 19:56:16.217031 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:16.216985    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb" podStartSLOduration=2.216969307 podStartE2EDuration="2.216969307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:16.212561151 +0000 UTC m=+1124.984854169" watchObservedRunningTime="2026-11-03 19:56:16.216969307 +0000 UTC m=+1124.989262344"
Nov 03 19:56:16.238466 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:16.238425    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd"]
Nov 03 19:56:16.244539 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:16.244514    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6cd7f4-9f64f859b-5mwmd"]
Nov 03 19:56:16.257676 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:16.257648    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb"]
Nov 03 19:56:16.268644 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:16.268615    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6cd7f4-7969c4c6bc-s6rgb"]
Nov 03 19:56:17.407978 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:17.407843    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08ebfbb6-454c-4aba-8c39-a64ec8878154" path="/var/lib/kubelet/pods/08ebfbb6-454c-4aba-8c39-a64ec8878154/volumes"
Nov 03 19:56:17.409397 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:17.409372    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8a22a57-ff2f-407d-8a50-90c8a26ec6ff" path="/var/lib/kubelet/pods/f8a22a57-ff2f-407d-8a50-90c8a26ec6ff/volumes"
Nov 03 19:56:18.016361 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:18.016335    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66"
Nov 03 19:56:22.536259 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:22.536227    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf"]
Nov 03 19:56:22.536906 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:22.536891    5591 cpu_manager.go:414] "containerMap: removing container" podUID="f8a22a57-ff2f-407d-8a50-90c8a26ec6ff" containerName="payload-processing"
Nov 03 19:56:22.536906 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:22.536906    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8a22a57-ff2f-407d-8a50-90c8a26ec6ff" containerName="payload-processing"
Nov 03 19:56:22.537051 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:22.536954    5591 cpu_manager.go:414] "containerMap: removing container" podUID="08ebfbb6-454c-4aba-8c39-a64ec8878154" containerName="payload-pre-processing"
Nov 03 19:56:22.537051 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:22.536959    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ebfbb6-454c-4aba-8c39-a64ec8878154" containerName="payload-pre-processing"
Nov 03 19:56:22.537159 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:22.537063    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ebfbb6-454c-4aba-8c39-a64ec8878154" containerName="payload-pre-processing"
Nov 03 19:56:22.537159 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:22.537073    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8a22a57-ff2f-407d-8a50-90c8a26ec6ff" containerName="payload-processing"
Nov 03 19:56:22.540994 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:22.540968    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf"
Nov 03 19:56:22.552624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:22.549285    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-b34f6c-serving-cert\""
Nov 03 19:56:22.556663 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:22.556350    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf"]
Nov 03 19:56:22.633727 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:22.633683    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-openshift-service-ca\") pod \"maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf\" (UID: \"3f2c38b0-cb8a-4fec-9630-be624f51ff9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf"
Nov 03 19:56:22.633880 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:22.633809    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lb54\" (UniqueName: \"kubernetes.io/projected/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-kube-api-access-9lb54\") pod \"maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf\" (UID: \"3f2c38b0-cb8a-4fec-9630-be624f51ff9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf"
Nov 03 19:56:22.633880 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:22.633876    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-metrics-certs\") pod \"maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf\" (UID: \"3f2c38b0-cb8a-4fec-9630-be624f51ff9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf"
Nov 03 19:56:22.634023 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:22.634007    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-maas-api-tls\") pod \"maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf\" (UID: \"3f2c38b0-cb8a-4fec-9630-be624f51ff9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf"
Nov 03 19:56:22.746230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:22.742378    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-maas-api-tls\") pod \"maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf\" (UID: \"3f2c38b0-cb8a-4fec-9630-be624f51ff9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf"
Nov 03 19:56:22.746230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:22.742580    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-openshift-service-ca\") pod \"maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf\" (UID: \"3f2c38b0-cb8a-4fec-9630-be624f51ff9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf"
Nov 03 19:56:22.746230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:22.742688    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lb54\" (UniqueName: \"kubernetes.io/projected/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-kube-api-access-9lb54\") pod \"maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf\" (UID: \"3f2c38b0-cb8a-4fec-9630-be624f51ff9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf"
Nov 03 19:56:22.746230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:22.742805    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-metrics-certs\") pod \"maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf\" (UID: \"3f2c38b0-cb8a-4fec-9630-be624f51ff9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf"
Nov 03 19:56:22.751226 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:22.748028    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-openshift-service-ca\") pod \"maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf\" (UID: \"3f2c38b0-cb8a-4fec-9630-be624f51ff9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf"
Nov 03 19:56:22.774621 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:22.752127    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-maas-api-tls\") pod \"maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf\" (UID: \"3f2c38b0-cb8a-4fec-9630-be624f51ff9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf"
Nov 03 19:56:22.774621 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:22.752279    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-metrics-certs\") pod \"maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf\" (UID: \"3f2c38b0-cb8a-4fec-9630-be624f51ff9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf"
Nov 03 19:56:22.795384 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:22.785910    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lb54\" (UniqueName: \"kubernetes.io/projected/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-kube-api-access-9lb54\") pod \"maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf\" (UID: \"3f2c38b0-cb8a-4fec-9630-be624f51ff9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf"
Nov 03 19:56:22.866373 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:22.866339    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf"
Nov 03 19:56:23.229283 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:23.227370    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf"]
Nov 03 19:56:23.302893 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:23.302827    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf" event={"ID":"3f2c38b0-cb8a-4fec-9630-be624f51ff9d","Type":"ContainerStarted","Data":"28794cd52864debdc72bdfda495447d01cdce2d8a3714eaeaa3b6c0140c964df"}
Nov 03 19:56:24.312422 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:24.312389    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf" event={"ID":"3f2c38b0-cb8a-4fec-9630-be624f51ff9d","Type":"ContainerStarted","Data":"a64756151782805657df1e21a12f61c11b34a9ac0c06643066c93197a7da9d14"}
Nov 03 19:56:24.312919 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:24.312573    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf"
Nov 03 19:56:24.334752 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:24.334707    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf" podStartSLOduration=1.785138882 podStartE2EDuration="2.334692782s" totalImagesPullingTime="549.5539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:24.333933906 +0000 UTC m=+1133.106226925" watchObservedRunningTime="2026-11-03 19:56:24.334692782 +0000 UTC m=+1133.106985801"
Nov 03 19:56:27.183863 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:27.183830    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb"
Nov 03 19:56:28.785995 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:56:28.785953    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:56:30.388358 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:30.388330    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf"
Nov 03 19:56:40.919369 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:40.919332    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh"]
Nov 03 19:56:40.935066 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:40.935033    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc"]
Nov 03 19:56:41.803931 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:41.803892    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:56:43.139898 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:43.139862    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t"]
Nov 03 19:56:43.151578 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:43.151536    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t"]
Nov 03 19:56:43.151578 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:43.151644    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t"
Nov 03 19:56:43.163089 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:43.162319    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3d1db1\""
Nov 03 19:56:43.165853 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:43.165823    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q"]
Nov 03 19:56:43.171305 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:43.171282    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q"
Nov 03 19:56:43.211232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:43.184511    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3d1db1-dockercfg-w4ngn\""
Nov 03 19:56:43.295232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:43.264541    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q"]
Nov 03 19:56:43.420138 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:43.415970    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h652l\" (UniqueName: \"kubernetes.io/projected/454eb87e-3359-46e5-86e3-36ea2e6f0663-kube-api-access-h652l\") pod \"payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t\" (UID: \"454eb87e-3359-46e5-86e3-36ea2e6f0663\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t"
Nov 03 19:56:43.420489 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:43.420464    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454eb87e-3359-46e5-86e3-36ea2e6f0663-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t\" (UID: \"454eb87e-3359-46e5-86e3-36ea2e6f0663\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t"
Nov 03 19:56:43.420624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:43.420609    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljrq5\" (UniqueName: \"kubernetes.io/projected/8618c3c2-38f6-45fd-9e7b-9be8db40da4e-kube-api-access-ljrq5\") pod \"payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q\" (UID: \"8618c3c2-38f6-45fd-9e7b-9be8db40da4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q"
Nov 03 19:56:43.420879 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:43.420859    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8618c3c2-38f6-45fd-9e7b-9be8db40da4e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q\" (UID: \"8618c3c2-38f6-45fd-9e7b-9be8db40da4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q"
Nov 03 19:56:43.574189 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:43.574151    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h652l\" (UniqueName: \"kubernetes.io/projected/454eb87e-3359-46e5-86e3-36ea2e6f0663-kube-api-access-h652l\") pod \"payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t\" (UID: \"454eb87e-3359-46e5-86e3-36ea2e6f0663\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t"
Nov 03 19:56:43.574384 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:43.574262    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454eb87e-3359-46e5-86e3-36ea2e6f0663-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t\" (UID: \"454eb87e-3359-46e5-86e3-36ea2e6f0663\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t"
Nov 03 19:56:43.574384 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:43.574292    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljrq5\" (UniqueName: \"kubernetes.io/projected/8618c3c2-38f6-45fd-9e7b-9be8db40da4e-kube-api-access-ljrq5\") pod \"payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q\" (UID: \"8618c3c2-38f6-45fd-9e7b-9be8db40da4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q"
Nov 03 19:56:43.574384 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:43.574374    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8618c3c2-38f6-45fd-9e7b-9be8db40da4e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q\" (UID: \"8618c3c2-38f6-45fd-9e7b-9be8db40da4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q"
Nov 03 19:56:43.575098 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:43.575070    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454eb87e-3359-46e5-86e3-36ea2e6f0663-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t\" (UID: \"454eb87e-3359-46e5-86e3-36ea2e6f0663\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t"
Nov 03 19:56:43.575220 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:43.575183    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8618c3c2-38f6-45fd-9e7b-9be8db40da4e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q\" (UID: \"8618c3c2-38f6-45fd-9e7b-9be8db40da4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q"
Nov 03 19:56:43.598012 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:43.597970    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljrq5\" (UniqueName: \"kubernetes.io/projected/8618c3c2-38f6-45fd-9e7b-9be8db40da4e-kube-api-access-ljrq5\") pod \"payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q\" (UID: \"8618c3c2-38f6-45fd-9e7b-9be8db40da4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q"
Nov 03 19:56:43.599163 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:43.599127    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h652l\" (UniqueName: \"kubernetes.io/projected/454eb87e-3359-46e5-86e3-36ea2e6f0663-kube-api-access-h652l\") pod \"payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t\" (UID: \"454eb87e-3359-46e5-86e3-36ea2e6f0663\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t"
Nov 03 19:56:43.771264 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:43.771241    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t"
Nov 03 19:56:43.823756 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:43.823720    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q"
Nov 03 19:56:44.076356 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:44.076334    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t"]
Nov 03 19:56:44.079710 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:56:44.079683    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454eb87e_3359_46e5_86e3_36ea2e6f0663.slice/crio-863d60453aa393518c5903b8e0fcb4b63042ea0f92d0d188ccb07b74c31c088b WatchSource:0}: Error finding container 863d60453aa393518c5903b8e0fcb4b63042ea0f92d0d188ccb07b74c31c088b: Status 404 returned error can't find the container with id 863d60453aa393518c5903b8e0fcb4b63042ea0f92d0d188ccb07b74c31c088b
Nov 03 19:56:44.174815 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:44.174790    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q"]
Nov 03 19:56:44.176755 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:56:44.176725    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8618c3c2_38f6_45fd_9e7b_9be8db40da4e.slice/crio-f583b94509529b801139d0ea18eab10fb3d77da698802762a6094a386432c1f7 WatchSource:0}: Error finding container f583b94509529b801139d0ea18eab10fb3d77da698802762a6094a386432c1f7: Status 404 returned error can't find the container with id f583b94509529b801139d0ea18eab10fb3d77da698802762a6094a386432c1f7
Nov 03 19:56:44.240283 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:44.240244    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj"]
Nov 03 19:56:44.245420 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:44.245395    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj"
Nov 03 19:56:44.257318 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:44.256495    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9b3619-dockercfg-m9jnz\""
Nov 03 19:56:44.264292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:44.264267    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj"]
Nov 03 19:56:44.338096 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:44.338012    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9b3619\""
Nov 03 19:56:44.408743 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:44.408240    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/685e1e6c-bc01-40af-beeb-8bbc2f218de5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj\" (UID: \"685e1e6c-bc01-40af-beeb-8bbc2f218de5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj"
Nov 03 19:56:44.408743 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:44.408499    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sblhz\" (UniqueName: \"kubernetes.io/projected/685e1e6c-bc01-40af-beeb-8bbc2f218de5-kube-api-access-sblhz\") pod \"payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj\" (UID: \"685e1e6c-bc01-40af-beeb-8bbc2f218de5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj"
Nov 03 19:56:44.517590 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:44.517549    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/685e1e6c-bc01-40af-beeb-8bbc2f218de5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj\" (UID: \"685e1e6c-bc01-40af-beeb-8bbc2f218de5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj"
Nov 03 19:56:44.518155 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:44.518121    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sblhz\" (UniqueName: \"kubernetes.io/projected/685e1e6c-bc01-40af-beeb-8bbc2f218de5-kube-api-access-sblhz\") pod \"payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj\" (UID: \"685e1e6c-bc01-40af-beeb-8bbc2f218de5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj"
Nov 03 19:56:44.520776 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:44.520742    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/685e1e6c-bc01-40af-beeb-8bbc2f218de5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj\" (UID: \"685e1e6c-bc01-40af-beeb-8bbc2f218de5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj"
Nov 03 19:56:44.545233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:44.543565    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sblhz\" (UniqueName: \"kubernetes.io/projected/685e1e6c-bc01-40af-beeb-8bbc2f218de5-kube-api-access-sblhz\") pod \"payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj\" (UID: \"685e1e6c-bc01-40af-beeb-8bbc2f218de5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj"
Nov 03 19:56:44.569124 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:44.569094    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj"
Nov 03 19:56:44.792370 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:44.792334    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q" event={"ID":"8618c3c2-38f6-45fd-9e7b-9be8db40da4e","Type":"ContainerStarted","Data":"a125616aae52771aaf2cda257d4dd54e7a8a055a328683cd11c4b695708b4ab0"}
Nov 03 19:56:44.792546 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:44.792397    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q" event={"ID":"8618c3c2-38f6-45fd-9e7b-9be8db40da4e","Type":"ContainerStarted","Data":"f583b94509529b801139d0ea18eab10fb3d77da698802762a6094a386432c1f7"}
Nov 03 19:56:44.792598 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:44.792571    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q"
Nov 03 19:56:44.794833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:44.794427    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t" event={"ID":"454eb87e-3359-46e5-86e3-36ea2e6f0663","Type":"ContainerStarted","Data":"de8eebc5c083a01ec9db3b8248706f4f1440c593a3fc9362b65fe076c022badf"}
Nov 03 19:56:44.794833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:44.794477    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t" event={"ID":"454eb87e-3359-46e5-86e3-36ea2e6f0663","Type":"ContainerStarted","Data":"863d60453aa393518c5903b8e0fcb4b63042ea0f92d0d188ccb07b74c31c088b"}
Nov 03 19:56:44.794833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:44.794519    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t"
Nov 03 19:56:44.818263 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:44.818155    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q" podStartSLOduration=1.818136314 podStartE2EDuration="1.818136314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:44.814009467 +0000 UTC m=+1153.586302488" watchObservedRunningTime="2026-11-03 19:56:44.818136314 +0000 UTC m=+1153.590429334"
Nov 03 19:56:44.840410 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:44.840310    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t" podStartSLOduration=2.840289597 podStartE2EDuration="2.840289597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:44.83583732 +0000 UTC m=+1153.608130336" watchObservedRunningTime="2026-11-03 19:56:44.840289597 +0000 UTC m=+1153.612582616"
Nov 03 19:56:45.043239 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:45.036541    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj"]
Nov 03 19:56:45.803965 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:45.803923    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj" event={"ID":"685e1e6c-bc01-40af-beeb-8bbc2f218de5","Type":"ContainerStarted","Data":"825a9526c87e44117f13cf7fa6b898b07d23a8796d03f1b33376033f9c53b37e"}
Nov 03 19:56:45.804488 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:45.804231    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj"
Nov 03 19:56:45.804488 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:45.804285    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj" event={"ID":"685e1e6c-bc01-40af-beeb-8bbc2f218de5","Type":"ContainerStarted","Data":"eb3b5524be0e2a03ecd0a414843d173cbdc214a1c125f73fc784e9af5f02427b"}
Nov 03 19:56:45.899532 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:45.899471    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj" podStartSLOduration=1.899452159 podStartE2EDuration="1.899452159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:45.893754132 +0000 UTC m=+1154.666047150" watchObservedRunningTime="2026-11-03 19:56:45.899452159 +0000 UTC m=+1154.671745178"
Nov 03 19:56:45.932095 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:45.921914    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh" podUID="fa147a82-270a-4322-aaf2-9f7a8e73b82b" containerName="payload-processing" containerID="cri-o://abeffea6e83b48ca6c0946967ff99f47ba7cd10cc3f34877e242278b996ee55a" gracePeriod=25
Nov 03 19:56:45.941304 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:45.937419    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc" podUID="76760b0b-7d3b-48a5-8c11-44efcec67ef1" containerName="payload-pre-processing" containerID="cri-o://2af9095de57ef271fa128861e068013f2097b51dc2a1ac33da09760734e71340" gracePeriod=25
Nov 03 19:56:46.608678 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.608648    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc"
Nov 03 19:56:46.662985 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.662913    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76760b0b-7d3b-48a5-8c11-44efcec67ef1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76760b0b-7d3b-48a5-8c11-44efcec67ef1-plugins-config-volume\") pod \"76760b0b-7d3b-48a5-8c11-44efcec67ef1\" (UID: \"76760b0b-7d3b-48a5-8c11-44efcec67ef1\") "
Nov 03 19:56:46.663174 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.663161    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76760b0b-7d3b-48a5-8c11-44efcec67ef1-kube-api-access-gcqpf\" (UniqueName: \"kubernetes.io/projected/76760b0b-7d3b-48a5-8c11-44efcec67ef1-kube-api-access-gcqpf\") pod \"76760b0b-7d3b-48a5-8c11-44efcec67ef1\" (UID: \"76760b0b-7d3b-48a5-8c11-44efcec67ef1\") "
Nov 03 19:56:46.663661 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.663636    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76760b0b-7d3b-48a5-8c11-44efcec67ef1-plugins-config-volume" pod "76760b0b-7d3b-48a5-8c11-44efcec67ef1" (UID: "76760b0b-7d3b-48a5-8c11-44efcec67ef1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:56:46.663808 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.663790    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76760b0b-7d3b-48a5-8c11-44efcec67ef1-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:46.665795 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.665774    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76760b0b-7d3b-48a5-8c11-44efcec67ef1-kube-api-access-gcqpf" pod "76760b0b-7d3b-48a5-8c11-44efcec67ef1" (UID: "76760b0b-7d3b-48a5-8c11-44efcec67ef1"). InnerVolumeSpecName "kube-api-access-gcqpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:56:46.670287 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.670266    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh"
Nov 03 19:56:46.791783 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.791739    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa147a82-270a-4322-aaf2-9f7a8e73b82b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa147a82-270a-4322-aaf2-9f7a8e73b82b-plugins-config-volume\") pod \"fa147a82-270a-4322-aaf2-9f7a8e73b82b\" (UID: \"fa147a82-270a-4322-aaf2-9f7a8e73b82b\") "
Nov 03 19:56:46.792170 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.792140    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa147a82-270a-4322-aaf2-9f7a8e73b82b-kube-api-access-28r48\" (UniqueName: \"kubernetes.io/projected/fa147a82-270a-4322-aaf2-9f7a8e73b82b-kube-api-access-28r48\") pod \"fa147a82-270a-4322-aaf2-9f7a8e73b82b\" (UID: \"fa147a82-270a-4322-aaf2-9f7a8e73b82b\") "
Nov 03 19:56:46.798898 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.793520    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcqpf\" (UniqueName: \"kubernetes.io/projected/76760b0b-7d3b-48a5-8c11-44efcec67ef1-kube-api-access-gcqpf\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:46.800163 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.800127    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa147a82-270a-4322-aaf2-9f7a8e73b82b-plugins-config-volume" pod "fa147a82-270a-4322-aaf2-9f7a8e73b82b" (UID: "fa147a82-270a-4322-aaf2-9f7a8e73b82b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:56:46.828013 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.827957    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa147a82-270a-4322-aaf2-9f7a8e73b82b-kube-api-access-28r48" pod "fa147a82-270a-4322-aaf2-9f7a8e73b82b" (UID: "fa147a82-270a-4322-aaf2-9f7a8e73b82b"). InnerVolumeSpecName "kube-api-access-28r48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:56:46.864194 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.864148    5591 generic.go:356] "Generic (PLEG): container finished" podID="76760b0b-7d3b-48a5-8c11-44efcec67ef1" containerID="2af9095de57ef271fa128861e068013f2097b51dc2a1ac33da09760734e71340" exitCode=0
Nov 03 19:56:46.864432 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.864292    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc"
Nov 03 19:56:46.864432 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.864324    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc" event={"ID":"76760b0b-7d3b-48a5-8c11-44efcec67ef1","Type":"ContainerDied","Data":"2af9095de57ef271fa128861e068013f2097b51dc2a1ac33da09760734e71340"}
Nov 03 19:56:46.864432 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.864384    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc" event={"ID":"76760b0b-7d3b-48a5-8c11-44efcec67ef1","Type":"ContainerDied","Data":"8b95ccc03ed82791855300a40935591471ddf25e82e96ff16551daee628fdb32"}
Nov 03 19:56:46.864432 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.864413    5591 scope.go:122] "RemoveContainer" containerID="2af9095de57ef271fa128861e068013f2097b51dc2a1ac33da09760734e71340"
Nov 03 19:56:46.867541 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.867517    5591 generic.go:356] "Generic (PLEG): container finished" podID="fa147a82-270a-4322-aaf2-9f7a8e73b82b" containerID="abeffea6e83b48ca6c0946967ff99f47ba7cd10cc3f34877e242278b996ee55a" exitCode=0
Nov 03 19:56:46.867656 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.867552    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh" event={"ID":"fa147a82-270a-4322-aaf2-9f7a8e73b82b","Type":"ContainerDied","Data":"abeffea6e83b48ca6c0946967ff99f47ba7cd10cc3f34877e242278b996ee55a"}
Nov 03 19:56:46.867656 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.867592    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh"
Nov 03 19:56:46.867656 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.867612    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh" event={"ID":"fa147a82-270a-4322-aaf2-9f7a8e73b82b","Type":"ContainerDied","Data":"fa9fd6e076c7b9c089f9508cc46fcc72bea6dc28f7c7cd7e5a82b537ddb6373b"}
Nov 03 19:56:46.877414 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.877389    5591 scope.go:122] "RemoveContainer" containerID="2af9095de57ef271fa128861e068013f2097b51dc2a1ac33da09760734e71340"
Nov 03 19:56:46.877700 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:56:46.877678    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2af9095de57ef271fa128861e068013f2097b51dc2a1ac33da09760734e71340\": container with ID starting with 2af9095de57ef271fa128861e068013f2097b51dc2a1ac33da09760734e71340 not found: ID does not exist" containerID="2af9095de57ef271fa128861e068013f2097b51dc2a1ac33da09760734e71340"
Nov 03 19:56:46.877749 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.877710    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2af9095de57ef271fa128861e068013f2097b51dc2a1ac33da09760734e71340"} err="failed to get container status \"2af9095de57ef271fa128861e068013f2097b51dc2a1ac33da09760734e71340\": rpc error: code = NotFound desc = could not find container \"2af9095de57ef271fa128861e068013f2097b51dc2a1ac33da09760734e71340\": container with ID starting with 2af9095de57ef271fa128861e068013f2097b51dc2a1ac33da09760734e71340 not found: ID does not exist"
Nov 03 19:56:46.877749 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.877725    5591 scope.go:122] "RemoveContainer" containerID="abeffea6e83b48ca6c0946967ff99f47ba7cd10cc3f34877e242278b996ee55a"
Nov 03 19:56:46.889302 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.889283    5591 scope.go:122] "RemoveContainer" containerID="abeffea6e83b48ca6c0946967ff99f47ba7cd10cc3f34877e242278b996ee55a"
Nov 03 19:56:46.889925 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:56:46.889900    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abeffea6e83b48ca6c0946967ff99f47ba7cd10cc3f34877e242278b996ee55a\": container with ID starting with abeffea6e83b48ca6c0946967ff99f47ba7cd10cc3f34877e242278b996ee55a not found: ID does not exist" containerID="abeffea6e83b48ca6c0946967ff99f47ba7cd10cc3f34877e242278b996ee55a"
Nov 03 19:56:46.890038 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.889934    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abeffea6e83b48ca6c0946967ff99f47ba7cd10cc3f34877e242278b996ee55a"} err="failed to get container status \"abeffea6e83b48ca6c0946967ff99f47ba7cd10cc3f34877e242278b996ee55a\": rpc error: code = NotFound desc = could not find container \"abeffea6e83b48ca6c0946967ff99f47ba7cd10cc3f34877e242278b996ee55a\": container with ID starting with abeffea6e83b48ca6c0946967ff99f47ba7cd10cc3f34877e242278b996ee55a not found: ID does not exist"
Nov 03 19:56:46.891422 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.891402    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc"]
Nov 03 19:56:46.898258 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.898238    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d8d238bf-575d65cbff-bk7cc"]
Nov 03 19:56:46.900404 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.900383    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa147a82-270a-4322-aaf2-9f7a8e73b82b-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:46.900483 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.900407    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28r48\" (UniqueName: \"kubernetes.io/projected/fa147a82-270a-4322-aaf2-9f7a8e73b82b-kube-api-access-28r48\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:46.914402 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.914356    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh"]
Nov 03 19:56:46.918557 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:46.918537    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d8d238bf-85c6b5bff4-t48bh"]
Nov 03 19:56:47.395004 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:47.393799    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76760b0b-7d3b-48a5-8c11-44efcec67ef1" path="/var/lib/kubelet/pods/76760b0b-7d3b-48a5-8c11-44efcec67ef1/volumes"
Nov 03 19:56:47.395004 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:47.394489    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa147a82-270a-4322-aaf2-9f7a8e73b82b" path="/var/lib/kubelet/pods/fa147a82-270a-4322-aaf2-9f7a8e73b82b/volumes"
Nov 03 19:56:49.215462 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:49.215427    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr"]
Nov 03 19:56:49.216162 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:49.215665    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr" podUID="fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481" containerName="maas-api" containerID="cri-o://c288a63656a4c99ed4973cb64c2952251a106911daf6b831a4bd4adf45e8c0e3" gracePeriod=30
Nov 03 19:56:49.802018 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:49.801988    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr"
Nov 03 19:56:49.893161 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:49.893125    5591 generic.go:356] "Generic (PLEG): container finished" podID="fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481" containerID="c288a63656a4c99ed4973cb64c2952251a106911daf6b831a4bd4adf45e8c0e3" exitCode=0
Nov 03 19:56:49.893373 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:49.893179    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr"
Nov 03 19:56:49.893373 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:49.893236    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr" event={"ID":"fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481","Type":"ContainerDied","Data":"c288a63656a4c99ed4973cb64c2952251a106911daf6b831a4bd4adf45e8c0e3"}
Nov 03 19:56:49.893373 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:49.893286    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr" event={"ID":"fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481","Type":"ContainerDied","Data":"4c58795cf308126c51294350ebb0bf57535f5ea33c6727d51a7aad3bdbff8e7a"}
Nov 03 19:56:49.893373 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:49.893311    5591 scope.go:122] "RemoveContainer" containerID="c288a63656a4c99ed4973cb64c2952251a106911daf6b831a4bd4adf45e8c0e3"
Nov 03 19:56:49.903558 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:49.903532    5591 scope.go:122] "RemoveContainer" containerID="c288a63656a4c99ed4973cb64c2952251a106911daf6b831a4bd4adf45e8c0e3"
Nov 03 19:56:49.903831 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:56:49.903811    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c288a63656a4c99ed4973cb64c2952251a106911daf6b831a4bd4adf45e8c0e3\": container with ID starting with c288a63656a4c99ed4973cb64c2952251a106911daf6b831a4bd4adf45e8c0e3 not found: ID does not exist" containerID="c288a63656a4c99ed4973cb64c2952251a106911daf6b831a4bd4adf45e8c0e3"
Nov 03 19:56:49.903877 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:49.903839    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c288a63656a4c99ed4973cb64c2952251a106911daf6b831a4bd4adf45e8c0e3"} err="failed to get container status \"c288a63656a4c99ed4973cb64c2952251a106911daf6b831a4bd4adf45e8c0e3\": rpc error: code = NotFound desc = could not find container \"c288a63656a4c99ed4973cb64c2952251a106911daf6b831a4bd4adf45e8c0e3\": container with ID starting with c288a63656a4c99ed4973cb64c2952251a106911daf6b831a4bd4adf45e8c0e3 not found: ID does not exist"
Nov 03 19:56:49.917419 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:49.917389    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-metrics-certs\") pod \"fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481\" (UID: \"fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481\") "
Nov 03 19:56:49.917541 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:49.917525    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-openshift-service-ca\") pod \"fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481\" (UID: \"fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481\") "
Nov 03 19:56:49.917617 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:49.917600    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-kube-api-access-xgnnn\" (UniqueName: \"kubernetes.io/projected/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-kube-api-access-xgnnn\") pod \"fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481\" (UID: \"fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481\") "
Nov 03 19:56:49.917677 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:49.917648    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-maas-api-tls\") pod \"fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481\" (UID: \"fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481\") "
Nov 03 19:56:49.917901 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:49.917877    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-openshift-service-ca" pod "fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481" (UID: "fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:56:49.918268 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:49.918250    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:49.919765 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:49.919745    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-kube-api-access-xgnnn" pod "fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481" (UID: "fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481"). InnerVolumeSpecName "kube-api-access-xgnnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:56:49.920220 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:49.920182    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-metrics-certs" pod "fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481" (UID: "fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:56:49.920377 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:49.920360    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-maas-api-tls" pod "fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481" (UID: "fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:56:50.023713 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:50.023625    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:50.023937 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:50.023920    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgnnn\" (UniqueName: \"kubernetes.io/projected/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-kube-api-access-xgnnn\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:50.024089 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:50.024017    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:50.224997 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:50.224968    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr"]
Nov 03 19:56:50.228372 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:50.228339    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d8d238bf-5bc9574d58-4wxpr"]
Nov 03 19:56:51.354876 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:51.354837    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481" path="/var/lib/kubelet/pods/fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481/volumes"
Nov 03 19:56:52.650191 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:52.650102    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v"]
Nov 03 19:56:52.651024 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:52.650997    5591 cpu_manager.go:414] "containerMap: removing container" podUID="fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481" containerName="maas-api"
Nov 03 19:56:52.651024 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:52.651019    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481" containerName="maas-api"
Nov 03 19:56:52.651285 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:52.651071    5591 cpu_manager.go:414] "containerMap: removing container" podUID="76760b0b-7d3b-48a5-8c11-44efcec67ef1" containerName="payload-pre-processing"
Nov 03 19:56:52.651285 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:52.651080    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76760b0b-7d3b-48a5-8c11-44efcec67ef1" containerName="payload-pre-processing"
Nov 03 19:56:52.651285 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:52.651119    5591 cpu_manager.go:414] "containerMap: removing container" podUID="fa147a82-270a-4322-aaf2-9f7a8e73b82b" containerName="payload-processing"
Nov 03 19:56:52.651285 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:52.651128    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa147a82-270a-4322-aaf2-9f7a8e73b82b" containerName="payload-processing"
Nov 03 19:56:52.652074 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:52.652053    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdce2d9f-ee57-4bc5-9d4d-9f8f8dda0481" containerName="maas-api"
Nov 03 19:56:52.652161 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:52.652087    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="76760b0b-7d3b-48a5-8c11-44efcec67ef1" containerName="payload-pre-processing"
Nov 03 19:56:52.652161 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:52.652104    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa147a82-270a-4322-aaf2-9f7a8e73b82b" containerName="payload-processing"
Nov 03 19:56:52.657007 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:52.656986    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v"
Nov 03 19:56:52.665367 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:52.665338    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-3d1db1-serving-cert\""
Nov 03 19:56:52.670783 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:52.670759    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v"]
Nov 03 19:56:52.812296 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:52.811425    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf807ba9-d308-4b56-96a4-702135b4d522-metrics-certs\") pod \"maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v\" (UID: \"bf807ba9-d308-4b56-96a4-702135b4d522\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v"
Nov 03 19:56:52.812296 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:52.811540    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf807ba9-d308-4b56-96a4-702135b4d522-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v\" (UID: \"bf807ba9-d308-4b56-96a4-702135b4d522\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v"
Nov 03 19:56:52.812296 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:52.811755    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jghjp\" (UniqueName: \"kubernetes.io/projected/bf807ba9-d308-4b56-96a4-702135b4d522-kube-api-access-jghjp\") pod \"maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v\" (UID: \"bf807ba9-d308-4b56-96a4-702135b4d522\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v"
Nov 03 19:56:52.812296 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:52.811838    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf807ba9-d308-4b56-96a4-702135b4d522-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v\" (UID: \"bf807ba9-d308-4b56-96a4-702135b4d522\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v"
Nov 03 19:56:52.912991 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:52.912908    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf807ba9-d308-4b56-96a4-702135b4d522-metrics-certs\") pod \"maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v\" (UID: \"bf807ba9-d308-4b56-96a4-702135b4d522\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v"
Nov 03 19:56:52.913161 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:52.912994    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf807ba9-d308-4b56-96a4-702135b4d522-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v\" (UID: \"bf807ba9-d308-4b56-96a4-702135b4d522\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v"
Nov 03 19:56:52.913161 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:52.913049    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jghjp\" (UniqueName: \"kubernetes.io/projected/bf807ba9-d308-4b56-96a4-702135b4d522-kube-api-access-jghjp\") pod \"maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v\" (UID: \"bf807ba9-d308-4b56-96a4-702135b4d522\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v"
Nov 03 19:56:52.913161 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:52.913109    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf807ba9-d308-4b56-96a4-702135b4d522-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v\" (UID: \"bf807ba9-d308-4b56-96a4-702135b4d522\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v"
Nov 03 19:56:52.914230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:52.914186    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf807ba9-d308-4b56-96a4-702135b4d522-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v\" (UID: \"bf807ba9-d308-4b56-96a4-702135b4d522\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v"
Nov 03 19:56:52.915812 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:52.915789    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf807ba9-d308-4b56-96a4-702135b4d522-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v\" (UID: \"bf807ba9-d308-4b56-96a4-702135b4d522\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v"
Nov 03 19:56:52.915901 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:52.915844    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf807ba9-d308-4b56-96a4-702135b4d522-metrics-certs\") pod \"maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v\" (UID: \"bf807ba9-d308-4b56-96a4-702135b4d522\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v"
Nov 03 19:56:52.925347 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:52.925320    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jghjp\" (UniqueName: \"kubernetes.io/projected/bf807ba9-d308-4b56-96a4-702135b4d522-kube-api-access-jghjp\") pod \"maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v\" (UID: \"bf807ba9-d308-4b56-96a4-702135b4d522\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v"
Nov 03 19:56:52.979634 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:52.979599    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v"
Nov 03 19:56:53.335841 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:53.335317    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v"]
Nov 03 19:56:53.337695 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:56:53.337669    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf807ba9_d308_4b56_96a4_702135b4d522.slice/crio-93abca9a463edd18471d41065b7b454288b94bcc58182c242319dbeb2f837728 WatchSource:0}: Error finding container 93abca9a463edd18471d41065b7b454288b94bcc58182c242319dbeb2f837728: Status 404 returned error can't find the container with id 93abca9a463edd18471d41065b7b454288b94bcc58182c242319dbeb2f837728
Nov 03 19:56:54.026589 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:54.026497    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v" event={"ID":"bf807ba9-d308-4b56-96a4-702135b4d522","Type":"ContainerStarted","Data":"93abca9a463edd18471d41065b7b454288b94bcc58182c242319dbeb2f837728"}
Nov 03 19:56:54.104682 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:54.104644    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh"]
Nov 03 19:56:54.223227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:54.222415    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6"]
Nov 03 19:56:55.041550 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:55.041520    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p"]
Nov 03 19:56:55.075225 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:55.067172    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v"
Nov 03 19:56:55.075225 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:55.067247    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p"]
Nov 03 19:56:55.075225 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:55.067286    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v" event={"ID":"bf807ba9-d308-4b56-96a4-702135b4d522","Type":"ContainerStarted","Data":"67258483a817d28c97e180d083219eafcc8ba25241810ac5a93741c370d810c3"}
Nov 03 19:56:55.075225 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:55.067374    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p"
Nov 03 19:56:55.079780 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:55.079680    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v" podStartSLOduration=2.560150548 podStartE2EDuration="3.079660487s" totalImagesPullingTime="519.509939ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:55.075668324 +0000 UTC m=+1163.847961412" watchObservedRunningTime="2026-11-03 19:56:55.079660487 +0000 UTC m=+1163.851953506"
Nov 03 19:56:55.092411 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:55.092382    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-9b3619-serving-cert\""
Nov 03 19:56:55.144645 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:55.144611    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac0550ab-2fdd-41f1-90d9-081287e63228-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p\" (UID: \"ac0550ab-2fdd-41f1-90d9-081287e63228\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p"
Nov 03 19:56:55.144835 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:55.144683    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0550ab-2fdd-41f1-90d9-081287e63228-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p\" (UID: \"ac0550ab-2fdd-41f1-90d9-081287e63228\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p"
Nov 03 19:56:55.144835 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:55.144805    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prqch\" (UniqueName: \"kubernetes.io/projected/ac0550ab-2fdd-41f1-90d9-081287e63228-kube-api-access-prqch\") pod \"maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p\" (UID: \"ac0550ab-2fdd-41f1-90d9-081287e63228\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p"
Nov 03 19:56:55.145089 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:55.145059    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac0550ab-2fdd-41f1-90d9-081287e63228-metrics-certs\") pod \"maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p\" (UID: \"ac0550ab-2fdd-41f1-90d9-081287e63228\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p"
Nov 03 19:56:55.263227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:55.261009    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac0550ab-2fdd-41f1-90d9-081287e63228-metrics-certs\") pod \"maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p\" (UID: \"ac0550ab-2fdd-41f1-90d9-081287e63228\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p"
Nov 03 19:56:55.263227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:55.261273    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac0550ab-2fdd-41f1-90d9-081287e63228-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p\" (UID: \"ac0550ab-2fdd-41f1-90d9-081287e63228\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p"
Nov 03 19:56:55.263227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:55.261333    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0550ab-2fdd-41f1-90d9-081287e63228-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p\" (UID: \"ac0550ab-2fdd-41f1-90d9-081287e63228\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p"
Nov 03 19:56:55.263227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:55.261399    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prqch\" (UniqueName: \"kubernetes.io/projected/ac0550ab-2fdd-41f1-90d9-081287e63228-kube-api-access-prqch\") pod \"maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p\" (UID: \"ac0550ab-2fdd-41f1-90d9-081287e63228\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p"
Nov 03 19:56:55.271237 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:56:55.269279    5591 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-serving-cert: secret "maas-api-e2e-ipp-b-9b3619-serving-cert" not found
Nov 03 19:56:55.271237 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:56:55.269357    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac0550ab-2fdd-41f1-90d9-081287e63228-maas-api-tls podName:ac0550ab-2fdd-41f1-90d9-081287e63228 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:55.769334392 +0000 UTC m=+1164.541627409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ac0550ab-2fdd-41f1-90d9-081287e63228-maas-api-tls") pod "maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p" (UID: "ac0550ab-2fdd-41f1-90d9-081287e63228") : secret "maas-api-e2e-ipp-b-9b3619-serving-cert" not found
Nov 03 19:56:55.271237 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:55.270231    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0550ab-2fdd-41f1-90d9-081287e63228-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p\" (UID: \"ac0550ab-2fdd-41f1-90d9-081287e63228\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p"
Nov 03 19:56:55.271237 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:55.270927    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac0550ab-2fdd-41f1-90d9-081287e63228-metrics-certs\") pod \"maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p\" (UID: \"ac0550ab-2fdd-41f1-90d9-081287e63228\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p"
Nov 03 19:56:55.291243 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:55.290777    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prqch\" (UniqueName: \"kubernetes.io/projected/ac0550ab-2fdd-41f1-90d9-081287e63228-kube-api-access-prqch\") pod \"maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p\" (UID: \"ac0550ab-2fdd-41f1-90d9-081287e63228\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p"
Nov 03 19:56:55.772661 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:55.772617    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac0550ab-2fdd-41f1-90d9-081287e63228-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p\" (UID: \"ac0550ab-2fdd-41f1-90d9-081287e63228\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p"
Nov 03 19:56:55.792267 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:55.779578    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac0550ab-2fdd-41f1-90d9-081287e63228-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p\" (UID: \"ac0550ab-2fdd-41f1-90d9-081287e63228\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p"
Nov 03 19:56:55.852440 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:55.852406    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t"
Nov 03 19:56:55.863230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:55.859454    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q"
Nov 03 19:56:56.004619 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:56.004587    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p"
Nov 03 19:56:56.548101 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:56:56.547846    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0550ab_2fdd_41f1_90d9_081287e63228.slice/crio-75e54eb831d28b4542bf171615565bcd3b67ee5ca6af3d5e7edffdd8c1dad308 WatchSource:0}: Error finding container 75e54eb831d28b4542bf171615565bcd3b67ee5ca6af3d5e7edffdd8c1dad308: Status 404 returned error can't find the container with id 75e54eb831d28b4542bf171615565bcd3b67ee5ca6af3d5e7edffdd8c1dad308
Nov 03 19:56:56.552300 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:56.552263    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p"]
Nov 03 19:56:56.893283 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:56.892389    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj"
Nov 03 19:56:57.064529 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:57.064490    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p" event={"ID":"ac0550ab-2fdd-41f1-90d9-081287e63228","Type":"ContainerStarted","Data":"75e54eb831d28b4542bf171615565bcd3b67ee5ca6af3d5e7edffdd8c1dad308"}
Nov 03 19:56:57.556065 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:57.555284    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln"]
Nov 03 19:56:57.560705 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:57.560676    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln"
Nov 03 19:56:57.567512 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:57.567463    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-9a6d4b-serving-cert\""
Nov 03 19:56:57.567774 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:57.567749    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79nhf\" (UniqueName: \"kubernetes.io/projected/44e4835d-2ca1-4b1f-bcec-4095b41b6420-kube-api-access-79nhf\") pod \"maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln\" (UID: \"44e4835d-2ca1-4b1f-bcec-4095b41b6420\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln"
Nov 03 19:56:57.567879 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:57.567821    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44e4835d-2ca1-4b1f-bcec-4095b41b6420-metrics-certs\") pod \"maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln\" (UID: \"44e4835d-2ca1-4b1f-bcec-4095b41b6420\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln"
Nov 03 19:56:57.567995 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:57.567976    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44e4835d-2ca1-4b1f-bcec-4095b41b6420-maas-api-tls\") pod \"maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln\" (UID: \"44e4835d-2ca1-4b1f-bcec-4095b41b6420\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln"
Nov 03 19:56:57.568099 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:57.568055    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44e4835d-2ca1-4b1f-bcec-4095b41b6420-openshift-service-ca\") pod \"maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln\" (UID: \"44e4835d-2ca1-4b1f-bcec-4095b41b6420\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln"
Nov 03 19:56:57.574334 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:57.574308    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln"]
Nov 03 19:56:57.684233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:57.682653    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79nhf\" (UniqueName: \"kubernetes.io/projected/44e4835d-2ca1-4b1f-bcec-4095b41b6420-kube-api-access-79nhf\") pod \"maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln\" (UID: \"44e4835d-2ca1-4b1f-bcec-4095b41b6420\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln"
Nov 03 19:56:57.684233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:57.682725    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44e4835d-2ca1-4b1f-bcec-4095b41b6420-metrics-certs\") pod \"maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln\" (UID: \"44e4835d-2ca1-4b1f-bcec-4095b41b6420\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln"
Nov 03 19:56:57.684233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:57.682913    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44e4835d-2ca1-4b1f-bcec-4095b41b6420-maas-api-tls\") pod \"maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln\" (UID: \"44e4835d-2ca1-4b1f-bcec-4095b41b6420\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln"
Nov 03 19:56:57.684233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:57.683018    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44e4835d-2ca1-4b1f-bcec-4095b41b6420-openshift-service-ca\") pod \"maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln\" (UID: \"44e4835d-2ca1-4b1f-bcec-4095b41b6420\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln"
Nov 03 19:56:57.684233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:57.683939    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44e4835d-2ca1-4b1f-bcec-4095b41b6420-openshift-service-ca\") pod \"maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln\" (UID: \"44e4835d-2ca1-4b1f-bcec-4095b41b6420\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln"
Nov 03 19:56:57.737085 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:57.707888    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44e4835d-2ca1-4b1f-bcec-4095b41b6420-maas-api-tls\") pod \"maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln\" (UID: \"44e4835d-2ca1-4b1f-bcec-4095b41b6420\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln"
Nov 03 19:56:57.737085 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:57.731585    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44e4835d-2ca1-4b1f-bcec-4095b41b6420-metrics-certs\") pod \"maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln\" (UID: \"44e4835d-2ca1-4b1f-bcec-4095b41b6420\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln"
Nov 03 19:56:57.744503 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:57.738656    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79nhf\" (UniqueName: \"kubernetes.io/projected/44e4835d-2ca1-4b1f-bcec-4095b41b6420-kube-api-access-79nhf\") pod \"maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln\" (UID: \"44e4835d-2ca1-4b1f-bcec-4095b41b6420\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln"
Nov 03 19:56:57.881957 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:57.881857    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln"
Nov 03 19:56:58.075024 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:58.074987    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p" event={"ID":"ac0550ab-2fdd-41f1-90d9-081287e63228","Type":"ContainerStarted","Data":"0723aadd93c8327e817ae8d96601b72b9c4cdc0145807df980f2fc8352f714ff"}
Nov 03 19:56:58.075245 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:58.075092    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p"
Nov 03 19:56:58.100699 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:58.100646    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p" podStartSLOduration=2.662376196 podStartE2EDuration="3.100631744s" totalImagesPullingTime="438.255548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:58.09741084 +0000 UTC m=+1166.869703859" watchObservedRunningTime="2026-11-03 19:56:58.100631744 +0000 UTC m=+1166.872924776"
Nov 03 19:56:58.172932 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:58.172907    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln"]
Nov 03 19:56:58.174779 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:56:58.174752    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e4835d_2ca1_4b1f_bcec_4095b41b6420.slice/crio-547f6927ce23f7a4491f22529b7f88f05055a7a4e07fd7e66b02c5c44809a972 WatchSource:0}: Error finding container 547f6927ce23f7a4491f22529b7f88f05055a7a4e07fd7e66b02c5c44809a972: Status 404 returned error can't find the container with id 547f6927ce23f7a4491f22529b7f88f05055a7a4e07fd7e66b02c5c44809a972
Nov 03 19:56:58.798875 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:56:58.798578    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:56:59.107984 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:59.107161    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh" podUID="3d797d10-ce29-4e3e-a675-8cfbef31bc1e" containerName="payload-processing" containerID="cri-o://c38516704804d3cb49f1bb63b0d474314b235ef7a9d96550540ec9e14a81ad5a" gracePeriod=25
Nov 03 19:56:59.152343 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:59.152306    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln" event={"ID":"44e4835d-2ca1-4b1f-bcec-4095b41b6420","Type":"ContainerStarted","Data":"547f6927ce23f7a4491f22529b7f88f05055a7a4e07fd7e66b02c5c44809a972"}
Nov 03 19:56:59.223139 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:59.223099    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6" podUID="2f9077db-522f-45f1-af84-383047748cb3" containerName="payload-pre-processing" containerID="cri-o://e3e8a2efc8d13b9b2c0f9dcdb29a3add10c9924c76c80e01444dad7578af77ce" gracePeriod=25
Nov 03 19:56:59.512154 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:59.512093    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6"
Nov 03 19:56:59.548217 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:59.548171    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh"
Nov 03 19:56:59.637056 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:59.636942    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f9077db-522f-45f1-af84-383047748cb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f9077db-522f-45f1-af84-383047748cb3-plugins-config-volume\") pod \"2f9077db-522f-45f1-af84-383047748cb3\" (UID: \"2f9077db-522f-45f1-af84-383047748cb3\") "
Nov 03 19:56:59.637519 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:59.637392    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f9077db-522f-45f1-af84-383047748cb3-plugins-config-volume" pod "2f9077db-522f-45f1-af84-383047748cb3" (UID: "2f9077db-522f-45f1-af84-383047748cb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:56:59.637632 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:59.637542    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f9077db-522f-45f1-af84-383047748cb3-kube-api-access-ptmg2\" (UniqueName: \"kubernetes.io/projected/2f9077db-522f-45f1-af84-383047748cb3-kube-api-access-ptmg2\") pod \"2f9077db-522f-45f1-af84-383047748cb3\" (UID: \"2f9077db-522f-45f1-af84-383047748cb3\") "
Nov 03 19:56:59.637632 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:59.637608    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d797d10-ce29-4e3e-a675-8cfbef31bc1e-kube-api-access-ld94t\" (UniqueName: \"kubernetes.io/projected/3d797d10-ce29-4e3e-a675-8cfbef31bc1e-kube-api-access-ld94t\") pod \"3d797d10-ce29-4e3e-a675-8cfbef31bc1e\" (UID: \"3d797d10-ce29-4e3e-a675-8cfbef31bc1e\") "
Nov 03 19:56:59.637735 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:59.637663    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d797d10-ce29-4e3e-a675-8cfbef31bc1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d797d10-ce29-4e3e-a675-8cfbef31bc1e-plugins-config-volume\") pod \"3d797d10-ce29-4e3e-a675-8cfbef31bc1e\" (UID: \"3d797d10-ce29-4e3e-a675-8cfbef31bc1e\") "
Nov 03 19:56:59.638100 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:59.638066    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d797d10-ce29-4e3e-a675-8cfbef31bc1e-plugins-config-volume" pod "3d797d10-ce29-4e3e-a675-8cfbef31bc1e" (UID: "3d797d10-ce29-4e3e-a675-8cfbef31bc1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:56:59.638604 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:59.638581    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f9077db-522f-45f1-af84-383047748cb3-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:59.638693 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:59.638612    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d797d10-ce29-4e3e-a675-8cfbef31bc1e-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:59.639864 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:59.639841    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f9077db-522f-45f1-af84-383047748cb3-kube-api-access-ptmg2" pod "2f9077db-522f-45f1-af84-383047748cb3" (UID: "2f9077db-522f-45f1-af84-383047748cb3"). InnerVolumeSpecName "kube-api-access-ptmg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:56:59.639962 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:59.639907    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d797d10-ce29-4e3e-a675-8cfbef31bc1e-kube-api-access-ld94t" pod "3d797d10-ce29-4e3e-a675-8cfbef31bc1e" (UID: "3d797d10-ce29-4e3e-a675-8cfbef31bc1e"). InnerVolumeSpecName "kube-api-access-ld94t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:56:59.779594 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:59.779556    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptmg2\" (UniqueName: \"kubernetes.io/projected/2f9077db-522f-45f1-af84-383047748cb3-kube-api-access-ptmg2\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:56:59.787228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:56:59.782247    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld94t\" (UniqueName: \"kubernetes.io/projected/3d797d10-ce29-4e3e-a675-8cfbef31bc1e-kube-api-access-ld94t\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:00.160416 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:00.160384    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln" event={"ID":"44e4835d-2ca1-4b1f-bcec-4095b41b6420","Type":"ContainerStarted","Data":"8ebc1e8052f16fb159e317b34b42ead1b212f594eb06f1447ad887c3b7c278cd"}
Nov 03 19:57:00.160850 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:00.160619    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln"
Nov 03 19:57:00.161882 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:00.161858    5591 generic.go:356] "Generic (PLEG): container finished" podID="3d797d10-ce29-4e3e-a675-8cfbef31bc1e" containerID="c38516704804d3cb49f1bb63b0d474314b235ef7a9d96550540ec9e14a81ad5a" exitCode=0
Nov 03 19:57:00.162023 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:00.161894    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh"
Nov 03 19:57:00.162023 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:00.161944    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh" event={"ID":"3d797d10-ce29-4e3e-a675-8cfbef31bc1e","Type":"ContainerDied","Data":"c38516704804d3cb49f1bb63b0d474314b235ef7a9d96550540ec9e14a81ad5a"}
Nov 03 19:57:00.162023 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:00.162001    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh" event={"ID":"3d797d10-ce29-4e3e-a675-8cfbef31bc1e","Type":"ContainerDied","Data":"f8233dc364fba7b21d8102483aced7559c7a6e2a93ce0f287a1bdceb6415a920"}
Nov 03 19:57:00.162184 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:00.162029    5591 scope.go:122] "RemoveContainer" containerID="c38516704804d3cb49f1bb63b0d474314b235ef7a9d96550540ec9e14a81ad5a"
Nov 03 19:57:00.163400 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:00.163379    5591 generic.go:356] "Generic (PLEG): container finished" podID="2f9077db-522f-45f1-af84-383047748cb3" containerID="e3e8a2efc8d13b9b2c0f9dcdb29a3add10c9924c76c80e01444dad7578af77ce" exitCode=0
Nov 03 19:57:00.163494 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:00.163407    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6" event={"ID":"2f9077db-522f-45f1-af84-383047748cb3","Type":"ContainerDied","Data":"e3e8a2efc8d13b9b2c0f9dcdb29a3add10c9924c76c80e01444dad7578af77ce"}
Nov 03 19:57:00.163494 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:00.163436    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6"
Nov 03 19:57:00.163494 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:00.163463    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6" event={"ID":"2f9077db-522f-45f1-af84-383047748cb3","Type":"ContainerDied","Data":"35733c95f53869c7b19b486e664f5ff215ddcb58081a4d533d761c427807fefb"}
Nov 03 19:57:00.174137 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:00.174120    5591 scope.go:122] "RemoveContainer" containerID="c38516704804d3cb49f1bb63b0d474314b235ef7a9d96550540ec9e14a81ad5a"
Nov 03 19:57:00.174482 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:57:00.174461    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c38516704804d3cb49f1bb63b0d474314b235ef7a9d96550540ec9e14a81ad5a\": container with ID starting with c38516704804d3cb49f1bb63b0d474314b235ef7a9d96550540ec9e14a81ad5a not found: ID does not exist" containerID="c38516704804d3cb49f1bb63b0d474314b235ef7a9d96550540ec9e14a81ad5a"
Nov 03 19:57:00.174547 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:00.174490    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c38516704804d3cb49f1bb63b0d474314b235ef7a9d96550540ec9e14a81ad5a"} err="failed to get container status \"c38516704804d3cb49f1bb63b0d474314b235ef7a9d96550540ec9e14a81ad5a\": rpc error: code = NotFound desc = could not find container \"c38516704804d3cb49f1bb63b0d474314b235ef7a9d96550540ec9e14a81ad5a\": container with ID starting with c38516704804d3cb49f1bb63b0d474314b235ef7a9d96550540ec9e14a81ad5a not found: ID does not exist"
Nov 03 19:57:00.174547 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:00.174506    5591 scope.go:122] "RemoveContainer" containerID="e3e8a2efc8d13b9b2c0f9dcdb29a3add10c9924c76c80e01444dad7578af77ce"
Nov 03 19:57:00.183251 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:00.183190    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln" podStartSLOduration=2.622523824 podStartE2EDuration="3.18317861s" totalImagesPullingTime="560.654786ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:00.181111775 +0000 UTC m=+1168.953404794" watchObservedRunningTime="2026-11-03 19:57:00.18317861 +0000 UTC m=+1168.955471628"
Nov 03 19:57:00.184519 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:00.184499    5591 scope.go:122] "RemoveContainer" containerID="e3e8a2efc8d13b9b2c0f9dcdb29a3add10c9924c76c80e01444dad7578af77ce"
Nov 03 19:57:00.185333 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:57:00.184840    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3e8a2efc8d13b9b2c0f9dcdb29a3add10c9924c76c80e01444dad7578af77ce\": container with ID starting with e3e8a2efc8d13b9b2c0f9dcdb29a3add10c9924c76c80e01444dad7578af77ce not found: ID does not exist" containerID="e3e8a2efc8d13b9b2c0f9dcdb29a3add10c9924c76c80e01444dad7578af77ce"
Nov 03 19:57:00.185333 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:00.184878    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3e8a2efc8d13b9b2c0f9dcdb29a3add10c9924c76c80e01444dad7578af77ce"} err="failed to get container status \"e3e8a2efc8d13b9b2c0f9dcdb29a3add10c9924c76c80e01444dad7578af77ce\": rpc error: code = NotFound desc = could not find container \"e3e8a2efc8d13b9b2c0f9dcdb29a3add10c9924c76c80e01444dad7578af77ce\": container with ID starting with e3e8a2efc8d13b9b2c0f9dcdb29a3add10c9924c76c80e01444dad7578af77ce not found: ID does not exist"
Nov 03 19:57:00.200453 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:00.200422    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh"]
Nov 03 19:57:00.206764 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:00.206745    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c5b820-57f67f9dcd-dpfvh"]
Nov 03 19:57:00.221654 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:00.221630    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6"]
Nov 03 19:57:00.225643 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:00.225618    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c5b820-6ff5f8944c-v5hs6"]
Nov 03 19:57:01.059725 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:01.059699    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v"
Nov 03 19:57:01.367590 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:01.367303    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f9077db-522f-45f1-af84-383047748cb3" path="/var/lib/kubelet/pods/2f9077db-522f-45f1-af84-383047748cb3/volumes"
Nov 03 19:57:01.368874 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:01.368838    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d797d10-ce29-4e3e-a675-8cfbef31bc1e" path="/var/lib/kubelet/pods/3d797d10-ce29-4e3e-a675-8cfbef31bc1e/volumes"
Nov 03 19:57:04.158722 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:04.158681    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p"
Nov 03 19:57:04.816804 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:04.816772    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr"]
Nov 03 19:57:04.817570 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:04.817554    5591 cpu_manager.go:414] "containerMap: removing container" podUID="3d797d10-ce29-4e3e-a675-8cfbef31bc1e" containerName="payload-processing"
Nov 03 19:57:04.817570 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:04.817570    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d797d10-ce29-4e3e-a675-8cfbef31bc1e" containerName="payload-processing"
Nov 03 19:57:04.817691 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:04.817594    5591 cpu_manager.go:414] "containerMap: removing container" podUID="2f9077db-522f-45f1-af84-383047748cb3" containerName="payload-pre-processing"
Nov 03 19:57:04.817691 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:04.817603    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f9077db-522f-45f1-af84-383047748cb3" containerName="payload-pre-processing"
Nov 03 19:57:04.817749 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:04.817738    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f9077db-522f-45f1-af84-383047748cb3" containerName="payload-pre-processing"
Nov 03 19:57:04.817779 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:04.817755    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d797d10-ce29-4e3e-a675-8cfbef31bc1e" containerName="payload-processing"
Nov 03 19:57:04.822842 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:04.822822    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr"
Nov 03 19:57:04.831702 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:04.831132    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-6a7dc9-dockercfg-6cmbr\""
Nov 03 19:57:04.833609 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:04.833580    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6a7dc9\""
Nov 03 19:57:04.836989 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:04.836964    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr"]
Nov 03 19:57:04.911083 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:04.908541    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwk89\" (UniqueName: \"kubernetes.io/projected/8d4d0a3c-0312-4063-b331-3735c26dcc4c-kube-api-access-hwk89\") pod \"payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr\" (UID: \"8d4d0a3c-0312-4063-b331-3735c26dcc4c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr"
Nov 03 19:57:04.911083 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:04.908612    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d4d0a3c-0312-4063-b331-3735c26dcc4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr\" (UID: \"8d4d0a3c-0312-4063-b331-3735c26dcc4c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr"
Nov 03 19:57:05.021229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:05.020954    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwk89\" (UniqueName: \"kubernetes.io/projected/8d4d0a3c-0312-4063-b331-3735c26dcc4c-kube-api-access-hwk89\") pod \"payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr\" (UID: \"8d4d0a3c-0312-4063-b331-3735c26dcc4c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr"
Nov 03 19:57:05.021229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:05.021016    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d4d0a3c-0312-4063-b331-3735c26dcc4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr\" (UID: \"8d4d0a3c-0312-4063-b331-3735c26dcc4c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr"
Nov 03 19:57:05.055233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:05.050576    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d4d0a3c-0312-4063-b331-3735c26dcc4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr\" (UID: \"8d4d0a3c-0312-4063-b331-3735c26dcc4c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr"
Nov 03 19:57:05.079136 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:05.079053    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwk89\" (UniqueName: \"kubernetes.io/projected/8d4d0a3c-0312-4063-b331-3735c26dcc4c-kube-api-access-hwk89\") pod \"payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr\" (UID: \"8d4d0a3c-0312-4063-b331-3735c26dcc4c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr"
Nov 03 19:57:05.164232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:05.159946    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr"
Nov 03 19:57:05.610052 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:05.610010    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr"]
Nov 03 19:57:05.610366 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:57:05.610323    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d4d0a3c_0312_4063_b331_3735c26dcc4c.slice/crio-0f11e166e185cfd6c1b5599a5eef6f5a4c5b011e693eec7ce82f39d499db6a21 WatchSource:0}: Error finding container 0f11e166e185cfd6c1b5599a5eef6f5a4c5b011e693eec7ce82f39d499db6a21: Status 404 returned error can't find the container with id 0f11e166e185cfd6c1b5599a5eef6f5a4c5b011e693eec7ce82f39d499db6a21
Nov 03 19:57:05.783953 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:05.783917    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79"]
Nov 03 19:57:05.784346 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:05.784161    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79" podUID="4e5befbb-f695-4fc3-9fd3-e9b8191fa235" containerName="maas-api" containerID="cri-o://c12665f0882a0b91a58165d03cf43fe4f328f604b06bda9aca9c2a18fe764102" gracePeriod=30
Nov 03 19:57:06.222533 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:06.210869    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln"
Nov 03 19:57:06.364278 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:06.361630    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr" event={"ID":"8d4d0a3c-0312-4063-b331-3735c26dcc4c","Type":"ContainerStarted","Data":"a6c8fc09e8dd7b17f9387400c7949ad52f2d61be1ad8cad715c1d7316dd424aa"}
Nov 03 19:57:06.364278 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:06.361681    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr" event={"ID":"8d4d0a3c-0312-4063-b331-3735c26dcc4c","Type":"ContainerStarted","Data":"0f11e166e185cfd6c1b5599a5eef6f5a4c5b011e693eec7ce82f39d499db6a21"}
Nov 03 19:57:06.364278 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:06.362720    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr"
Nov 03 19:57:06.365711 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:06.365686    5591 generic.go:356] "Generic (PLEG): container finished" podID="4e5befbb-f695-4fc3-9fd3-e9b8191fa235" containerID="c12665f0882a0b91a58165d03cf43fe4f328f604b06bda9aca9c2a18fe764102" exitCode=0
Nov 03 19:57:06.365839 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:06.365783    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79" event={"ID":"4e5befbb-f695-4fc3-9fd3-e9b8191fa235","Type":"ContainerDied","Data":"c12665f0882a0b91a58165d03cf43fe4f328f604b06bda9aca9c2a18fe764102"}
Nov 03 19:57:06.385184 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:06.382258    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr" podStartSLOduration=2.382238371 podStartE2EDuration="2.382238371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:06.380610932 +0000 UTC m=+1175.152903951" watchObservedRunningTime="2026-11-03 19:57:06.382238371 +0000 UTC m=+1175.154531390"
Nov 03 19:57:06.402847 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:06.402822    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79"
Nov 03 19:57:06.456910 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:06.456877    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-metrics-certs\") pod \"4e5befbb-f695-4fc3-9fd3-e9b8191fa235\" (UID: \"4e5befbb-f695-4fc3-9fd3-e9b8191fa235\") "
Nov 03 19:57:06.457065 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:06.456945    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-openshift-service-ca\") pod \"4e5befbb-f695-4fc3-9fd3-e9b8191fa235\" (UID: \"4e5befbb-f695-4fc3-9fd3-e9b8191fa235\") "
Nov 03 19:57:06.457124 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:06.457093    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-maas-api-tls\") pod \"4e5befbb-f695-4fc3-9fd3-e9b8191fa235\" (UID: \"4e5befbb-f695-4fc3-9fd3-e9b8191fa235\") "
Nov 03 19:57:06.457248 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:06.457234    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-kube-api-access-5nhvx\" (UniqueName: \"kubernetes.io/projected/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-kube-api-access-5nhvx\") pod \"4e5befbb-f695-4fc3-9fd3-e9b8191fa235\" (UID: \"4e5befbb-f695-4fc3-9fd3-e9b8191fa235\") "
Nov 03 19:57:06.457362 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:06.457333    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-openshift-service-ca" pod "4e5befbb-f695-4fc3-9fd3-e9b8191fa235" (UID: "4e5befbb-f695-4fc3-9fd3-e9b8191fa235"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:57:06.457883 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:06.457860    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:06.459513 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:06.459486    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-metrics-certs" pod "4e5befbb-f695-4fc3-9fd3-e9b8191fa235" (UID: "4e5befbb-f695-4fc3-9fd3-e9b8191fa235"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:57:06.459513 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:06.459506    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-maas-api-tls" pod "4e5befbb-f695-4fc3-9fd3-e9b8191fa235" (UID: "4e5befbb-f695-4fc3-9fd3-e9b8191fa235"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:57:06.459657 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:06.459525    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-kube-api-access-5nhvx" pod "4e5befbb-f695-4fc3-9fd3-e9b8191fa235" (UID: "4e5befbb-f695-4fc3-9fd3-e9b8191fa235"). InnerVolumeSpecName "kube-api-access-5nhvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:57:06.588228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:06.588041    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nhvx\" (UniqueName: \"kubernetes.io/projected/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-kube-api-access-5nhvx\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:06.588228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:06.588077    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:06.588228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:06.588092    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e5befbb-f695-4fc3-9fd3-e9b8191fa235-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:07.426355 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:07.426321    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79"
Nov 03 19:57:07.426355 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:07.426346    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79" event={"ID":"4e5befbb-f695-4fc3-9fd3-e9b8191fa235","Type":"ContainerDied","Data":"48ab9a7802e4460b9435138278ebcc69a6de29e44e82a4868249d7b0598644b5"}
Nov 03 19:57:07.427036 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:07.426417    5591 scope.go:122] "RemoveContainer" containerID="c12665f0882a0b91a58165d03cf43fe4f328f604b06bda9aca9c2a18fe764102"
Nov 03 19:57:07.474008 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:07.473956    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79"]
Nov 03 19:57:07.478026 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:07.477999    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c5b820-cbf9886f8-spx79"]
Nov 03 19:57:08.960040 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:08.960007    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"]
Nov 03 19:57:08.960671 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:08.960343    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr" podUID="b4bee720-61a7-4d49-918d-2cec14c41bc0" containerName="istio-proxy" containerID="cri-o://42851fddf9898ce74631413f1755b6ec03027b177f6863f9bafddc4024b1cb12" gracePeriod=30
Nov 03 19:57:09.360238 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:09.359437    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e5befbb-f695-4fc3-9fd3-e9b8191fa235" path="/var/lib/kubelet/pods/4e5befbb-f695-4fc3-9fd3-e9b8191fa235/volumes"
Nov 03 19:57:11.524443 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:11.524353    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:57:12.056429 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:12.056392    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q"]
Nov 03 19:57:12.059336 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:12.059307    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t"]
Nov 03 19:57:13.067329 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:13.067291    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj"]
Nov 03 19:57:14.266235 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.266195    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:57:14.394402 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.394374    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-envoy\") pod \"b4bee720-61a7-4d49-918d-2cec14c41bc0\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") "
Nov 03 19:57:14.394560 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.394453    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-token\" (UniqueName: \"kubernetes.io/projected/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-token\") pod \"b4bee720-61a7-4d49-918d-2cec14c41bc0\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") "
Nov 03 19:57:14.394560 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.394480    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-data\") pod \"b4bee720-61a7-4d49-918d-2cec14c41bc0\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") "
Nov 03 19:57:14.394560 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.394506    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-workload-socket\") pod \"b4bee720-61a7-4d49-918d-2cec14c41bc0\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") "
Nov 03 19:57:14.394560 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.394527    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4bee720-61a7-4d49-918d-2cec14c41bc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4bee720-61a7-4d49-918d-2cec14c41bc0-istiod-ca-cert\") pod \"b4bee720-61a7-4d49-918d-2cec14c41bc0\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") "
Nov 03 19:57:14.394790 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.394603    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-podinfo\") pod \"b4bee720-61a7-4d49-918d-2cec14c41bc0\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") "
Nov 03 19:57:14.394790 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.394758    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-credential-socket\") pod \"b4bee720-61a7-4d49-918d-2cec14c41bc0\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") "
Nov 03 19:57:14.394898 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.394801    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4bee720-61a7-4d49-918d-2cec14c41bc0-kube-api-access-x99l5\" (UniqueName: \"kubernetes.io/projected/b4bee720-61a7-4d49-918d-2cec14c41bc0-kube-api-access-x99l5\") pod \"b4bee720-61a7-4d49-918d-2cec14c41bc0\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") "
Nov 03 19:57:14.394898 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.394843    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-data" pod "b4bee720-61a7-4d49-918d-2cec14c41bc0" (UID: "b4bee720-61a7-4d49-918d-2cec14c41bc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:57:14.395005 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.394898    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-workload-certs\") pod \"b4bee720-61a7-4d49-918d-2cec14c41bc0\" (UID: \"b4bee720-61a7-4d49-918d-2cec14c41bc0\") "
Nov 03 19:57:14.395255 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.394853    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-workload-socket" pod "b4bee720-61a7-4d49-918d-2cec14c41bc0" (UID: "b4bee720-61a7-4d49-918d-2cec14c41bc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:57:14.395255 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.395068    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-credential-socket" pod "b4bee720-61a7-4d49-918d-2cec14c41bc0" (UID: "b4bee720-61a7-4d49-918d-2cec14c41bc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:57:14.395255 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.395094    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4bee720-61a7-4d49-918d-2cec14c41bc0-istiod-ca-cert" pod "b4bee720-61a7-4d49-918d-2cec14c41bc0" (UID: "b4bee720-61a7-4d49-918d-2cec14c41bc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:57:14.395677 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.395260    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-workload-certs" pod "b4bee720-61a7-4d49-918d-2cec14c41bc0" (UID: "b4bee720-61a7-4d49-918d-2cec14c41bc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:57:14.396006 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.395982    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:14.396085 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.396012    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:14.396085 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.396028    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4bee720-61a7-4d49-918d-2cec14c41bc0-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:14.396085 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.396042    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:14.396085 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.396057    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:14.396857 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.396833    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-token" pod "b4bee720-61a7-4d49-918d-2cec14c41bc0" (UID: "b4bee720-61a7-4d49-918d-2cec14c41bc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:57:14.396952 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.396850    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-podinfo" pod "b4bee720-61a7-4d49-918d-2cec14c41bc0" (UID: "b4bee720-61a7-4d49-918d-2cec14c41bc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:57:14.396952 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.396866    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-envoy" pod "b4bee720-61a7-4d49-918d-2cec14c41bc0" (UID: "b4bee720-61a7-4d49-918d-2cec14c41bc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:57:14.397236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.397218    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4bee720-61a7-4d49-918d-2cec14c41bc0-kube-api-access-x99l5" pod "b4bee720-61a7-4d49-918d-2cec14c41bc0" (UID: "b4bee720-61a7-4d49-918d-2cec14c41bc0"). InnerVolumeSpecName "kube-api-access-x99l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:57:14.497899 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.497866    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:14.497899 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.497899    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x99l5\" (UniqueName: \"kubernetes.io/projected/b4bee720-61a7-4d49-918d-2cec14c41bc0-kube-api-access-x99l5\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:14.498091 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.497925    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:14.498091 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.497941    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4bee720-61a7-4d49-918d-2cec14c41bc0-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:14.670821 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.670726    5591 generic.go:356] "Generic (PLEG): container finished" podID="b4bee720-61a7-4d49-918d-2cec14c41bc0" containerID="42851fddf9898ce74631413f1755b6ec03027b177f6863f9bafddc4024b1cb12" exitCode=0
Nov 03 19:57:14.671085 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.671061    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"
Nov 03 19:57:14.671462 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.671061    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr" event={"ID":"b4bee720-61a7-4d49-918d-2cec14c41bc0","Type":"ContainerDied","Data":"42851fddf9898ce74631413f1755b6ec03027b177f6863f9bafddc4024b1cb12"}
Nov 03 19:57:14.671573 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.671522    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr" event={"ID":"b4bee720-61a7-4d49-918d-2cec14c41bc0","Type":"ContainerDied","Data":"f8a962c252a8b5c7ff6b213bb72a659cd652f65172bd3a15475700c51a4d6733"}
Nov 03 19:57:14.671573 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.671559    5591 scope.go:122] "RemoveContainer" containerID="42851fddf9898ce74631413f1755b6ec03027b177f6863f9bafddc4024b1cb12"
Nov 03 19:57:14.692821 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.692800    5591 scope.go:122] "RemoveContainer" containerID="2bb3323200c3e0065d8378450b6320683108c773e62b9185daabf984923934a8"
Nov 03 19:57:14.703845 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.703824    5591 scope.go:122] "RemoveContainer" containerID="42851fddf9898ce74631413f1755b6ec03027b177f6863f9bafddc4024b1cb12"
Nov 03 19:57:14.704264 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:57:14.704199    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42851fddf9898ce74631413f1755b6ec03027b177f6863f9bafddc4024b1cb12\": container with ID starting with 42851fddf9898ce74631413f1755b6ec03027b177f6863f9bafddc4024b1cb12 not found: ID does not exist" containerID="42851fddf9898ce74631413f1755b6ec03027b177f6863f9bafddc4024b1cb12"
Nov 03 19:57:14.704448 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.704276    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42851fddf9898ce74631413f1755b6ec03027b177f6863f9bafddc4024b1cb12"} err="failed to get container status \"42851fddf9898ce74631413f1755b6ec03027b177f6863f9bafddc4024b1cb12\": rpc error: code = NotFound desc = could not find container \"42851fddf9898ce74631413f1755b6ec03027b177f6863f9bafddc4024b1cb12\": container with ID starting with 42851fddf9898ce74631413f1755b6ec03027b177f6863f9bafddc4024b1cb12 not found: ID does not exist"
Nov 03 19:57:14.704448 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.704304    5591 scope.go:122] "RemoveContainer" containerID="2bb3323200c3e0065d8378450b6320683108c773e62b9185daabf984923934a8"
Nov 03 19:57:14.704872 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:57:14.704840    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bb3323200c3e0065d8378450b6320683108c773e62b9185daabf984923934a8\": container with ID starting with 2bb3323200c3e0065d8378450b6320683108c773e62b9185daabf984923934a8 not found: ID does not exist" containerID="2bb3323200c3e0065d8378450b6320683108c773e62b9185daabf984923934a8"
Nov 03 19:57:14.704989 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.704880    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bb3323200c3e0065d8378450b6320683108c773e62b9185daabf984923934a8"} err="failed to get container status \"2bb3323200c3e0065d8378450b6320683108c773e62b9185daabf984923934a8\": rpc error: code = NotFound desc = could not find container \"2bb3323200c3e0065d8378450b6320683108c773e62b9185daabf984923934a8\": container with ID starting with 2bb3323200c3e0065d8378450b6320683108c773e62b9185daabf984923934a8 not found: ID does not exist"
Nov 03 19:57:14.708890 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.708863    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"]
Nov 03 19:57:14.714183 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:14.714162    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c5b820-openshift-default-6c8f597f85-5zvzr"]
Nov 03 19:57:15.368196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:15.367685    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4bee720-61a7-4d49-918d-2cec14c41bc0" path="/var/lib/kubelet/pods/b4bee720-61a7-4d49-918d-2cec14c41bc0/volumes"
Nov 03 19:57:16.992712 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:16.992670    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb"]
Nov 03 19:57:16.993765 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:16.993740    5591 cpu_manager.go:414] "containerMap: removing container" podUID="b4bee720-61a7-4d49-918d-2cec14c41bc0" containerName="istio-proxy"
Nov 03 19:57:16.993765 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:16.993762    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4bee720-61a7-4d49-918d-2cec14c41bc0" containerName="istio-proxy"
Nov 03 19:57:16.993993 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:16.993777    5591 cpu_manager.go:414] "containerMap: removing container" podUID="4e5befbb-f695-4fc3-9fd3-e9b8191fa235" containerName="maas-api"
Nov 03 19:57:16.993993 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:16.993814    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e5befbb-f695-4fc3-9fd3-e9b8191fa235" containerName="maas-api"
Nov 03 19:57:16.993993 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:16.993843    5591 cpu_manager.go:414] "containerMap: removing container" podUID="b4bee720-61a7-4d49-918d-2cec14c41bc0" containerName="istio-proxy"
Nov 03 19:57:16.993993 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:16.993858    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4bee720-61a7-4d49-918d-2cec14c41bc0" containerName="istio-proxy"
Nov 03 19:57:16.994162 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:16.994037    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4bee720-61a7-4d49-918d-2cec14c41bc0" containerName="istio-proxy"
Nov 03 19:57:16.994162 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:16.994070    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4bee720-61a7-4d49-918d-2cec14c41bc0" containerName="istio-proxy"
Nov 03 19:57:16.994162 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:16.994077    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e5befbb-f695-4fc3-9fd3-e9b8191fa235" containerName="maas-api"
Nov 03 19:57:16.999505 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:16.999469    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb"
Nov 03 19:57:17.041859 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.041824    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb"]
Nov 03 19:57:17.057024 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.056966    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q" podUID="8618c3c2-38f6-45fd-9e7b-9be8db40da4e" containerName="payload-processing" containerID="cri-o://a125616aae52771aaf2cda257d4dd54e7a8a055a328683cd11c4b695708b4ab0" gracePeriod=25
Nov 03 19:57:17.060553 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.060497    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t" podUID="454eb87e-3359-46e5-86e3-36ea2e6f0663" containerName="payload-pre-processing" containerID="cri-o://de8eebc5c083a01ec9db3b8248706f4f1440c593a3fc9362b65fe076c022badf" gracePeriod=25
Nov 03 19:57:17.066600 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.066580    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-6a7dc9-serving-cert\""
Nov 03 19:57:17.166231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.159651    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ce9e626-faa5-4b01-8849-971bc8def06c-maas-api-tls\") pod \"maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb\" (UID: \"3ce9e626-faa5-4b01-8849-971bc8def06c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb"
Nov 03 19:57:17.166231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.159779    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ce9e626-faa5-4b01-8849-971bc8def06c-metrics-certs\") pod \"maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb\" (UID: \"3ce9e626-faa5-4b01-8849-971bc8def06c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb"
Nov 03 19:57:17.166231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.159840    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x626c\" (UniqueName: \"kubernetes.io/projected/3ce9e626-faa5-4b01-8849-971bc8def06c-kube-api-access-x626c\") pod \"maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb\" (UID: \"3ce9e626-faa5-4b01-8849-971bc8def06c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb"
Nov 03 19:57:17.166231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.160091    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce9e626-faa5-4b01-8849-971bc8def06c-openshift-service-ca\") pod \"maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb\" (UID: \"3ce9e626-faa5-4b01-8849-971bc8def06c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb"
Nov 03 19:57:17.263992 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.262257    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x626c\" (UniqueName: \"kubernetes.io/projected/3ce9e626-faa5-4b01-8849-971bc8def06c-kube-api-access-x626c\") pod \"maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb\" (UID: \"3ce9e626-faa5-4b01-8849-971bc8def06c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb"
Nov 03 19:57:17.263992 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.262493    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce9e626-faa5-4b01-8849-971bc8def06c-openshift-service-ca\") pod \"maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb\" (UID: \"3ce9e626-faa5-4b01-8849-971bc8def06c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb"
Nov 03 19:57:17.264226 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.264123    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ce9e626-faa5-4b01-8849-971bc8def06c-maas-api-tls\") pod \"maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb\" (UID: \"3ce9e626-faa5-4b01-8849-971bc8def06c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb"
Nov 03 19:57:17.264289 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.264237    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ce9e626-faa5-4b01-8849-971bc8def06c-metrics-certs\") pod \"maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb\" (UID: \"3ce9e626-faa5-4b01-8849-971bc8def06c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb"
Nov 03 19:57:17.264289 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.264488    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce9e626-faa5-4b01-8849-971bc8def06c-openshift-service-ca\") pod \"maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb\" (UID: \"3ce9e626-faa5-4b01-8849-971bc8def06c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb"
Nov 03 19:57:17.268268 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.267654    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ce9e626-faa5-4b01-8849-971bc8def06c-maas-api-tls\") pod \"maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb\" (UID: \"3ce9e626-faa5-4b01-8849-971bc8def06c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb"
Nov 03 19:57:17.277361 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.277332    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ce9e626-faa5-4b01-8849-971bc8def06c-metrics-certs\") pod \"maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb\" (UID: \"3ce9e626-faa5-4b01-8849-971bc8def06c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb"
Nov 03 19:57:17.290434 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.290371    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x626c\" (UniqueName: \"kubernetes.io/projected/3ce9e626-faa5-4b01-8849-971bc8def06c-kube-api-access-x626c\") pod \"maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb\" (UID: \"3ce9e626-faa5-4b01-8849-971bc8def06c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb"
Nov 03 19:57:17.330050 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.329529    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb"
Nov 03 19:57:17.440124 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.440099    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q"
Nov 03 19:57:17.505282 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.505256    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t"
Nov 03 19:57:17.569281 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.569242    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8618c3c2-38f6-45fd-9e7b-9be8db40da4e-kube-api-access-ljrq5\" (UniqueName: \"kubernetes.io/projected/8618c3c2-38f6-45fd-9e7b-9be8db40da4e-kube-api-access-ljrq5\") pod \"8618c3c2-38f6-45fd-9e7b-9be8db40da4e\" (UID: \"8618c3c2-38f6-45fd-9e7b-9be8db40da4e\") "
Nov 03 19:57:17.569571 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.569549    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8618c3c2-38f6-45fd-9e7b-9be8db40da4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8618c3c2-38f6-45fd-9e7b-9be8db40da4e-plugins-config-volume\") pod \"8618c3c2-38f6-45fd-9e7b-9be8db40da4e\" (UID: \"8618c3c2-38f6-45fd-9e7b-9be8db40da4e\") "
Nov 03 19:57:17.569994 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.569939    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8618c3c2-38f6-45fd-9e7b-9be8db40da4e-plugins-config-volume" pod "8618c3c2-38f6-45fd-9e7b-9be8db40da4e" (UID: "8618c3c2-38f6-45fd-9e7b-9be8db40da4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:57:17.570317 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.570297    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8618c3c2-38f6-45fd-9e7b-9be8db40da4e-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:17.572136 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.572104    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8618c3c2-38f6-45fd-9e7b-9be8db40da4e-kube-api-access-ljrq5" pod "8618c3c2-38f6-45fd-9e7b-9be8db40da4e" (UID: "8618c3c2-38f6-45fd-9e7b-9be8db40da4e"). InnerVolumeSpecName "kube-api-access-ljrq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:57:17.646569 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.646543    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb"]
Nov 03 19:57:17.647269 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:57:17.647241    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce9e626_faa5_4b01_8849_971bc8def06c.slice/crio-58892f7915d1785b67a58285bc71c2aa057c2cfc41bf363e4f47c23f18f863ec WatchSource:0}: Error finding container 58892f7915d1785b67a58285bc71c2aa057c2cfc41bf363e4f47c23f18f863ec: Status 404 returned error can't find the container with id 58892f7915d1785b67a58285bc71c2aa057c2cfc41bf363e4f47c23f18f863ec
Nov 03 19:57:17.741061 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.741022    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/454eb87e-3359-46e5-86e3-36ea2e6f0663-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454eb87e-3359-46e5-86e3-36ea2e6f0663-plugins-config-volume\") pod \"454eb87e-3359-46e5-86e3-36ea2e6f0663\" (UID: \"454eb87e-3359-46e5-86e3-36ea2e6f0663\") "
Nov 03 19:57:17.741496 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.741473    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/454eb87e-3359-46e5-86e3-36ea2e6f0663-kube-api-access-h652l\" (UniqueName: \"kubernetes.io/projected/454eb87e-3359-46e5-86e3-36ea2e6f0663-kube-api-access-h652l\") pod \"454eb87e-3359-46e5-86e3-36ea2e6f0663\" (UID: \"454eb87e-3359-46e5-86e3-36ea2e6f0663\") "
Nov 03 19:57:17.773703 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.773670    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljrq5\" (UniqueName: \"kubernetes.io/projected/8618c3c2-38f6-45fd-9e7b-9be8db40da4e-kube-api-access-ljrq5\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:17.774156 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.767908    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/454eb87e-3359-46e5-86e3-36ea2e6f0663-plugins-config-volume" pod "454eb87e-3359-46e5-86e3-36ea2e6f0663" (UID: "454eb87e-3359-46e5-86e3-36ea2e6f0663"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:57:17.790644 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.785469    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/454eb87e-3359-46e5-86e3-36ea2e6f0663-kube-api-access-h652l" pod "454eb87e-3359-46e5-86e3-36ea2e6f0663" (UID: "454eb87e-3359-46e5-86e3-36ea2e6f0663"). InnerVolumeSpecName "kube-api-access-h652l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:57:17.790644 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.786825    5591 generic.go:356] "Generic (PLEG): container finished" podID="8618c3c2-38f6-45fd-9e7b-9be8db40da4e" containerID="a125616aae52771aaf2cda257d4dd54e7a8a055a328683cd11c4b695708b4ab0" exitCode=0
Nov 03 19:57:17.790644 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.786913    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q" event={"ID":"8618c3c2-38f6-45fd-9e7b-9be8db40da4e","Type":"ContainerDied","Data":"a125616aae52771aaf2cda257d4dd54e7a8a055a328683cd11c4b695708b4ab0"}
Nov 03 19:57:17.790644 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.786952    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q" event={"ID":"8618c3c2-38f6-45fd-9e7b-9be8db40da4e","Type":"ContainerDied","Data":"f583b94509529b801139d0ea18eab10fb3d77da698802762a6094a386432c1f7"}
Nov 03 19:57:17.790644 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.786974    5591 scope.go:122] "RemoveContainer" containerID="a125616aae52771aaf2cda257d4dd54e7a8a055a328683cd11c4b695708b4ab0"
Nov 03 19:57:17.790644 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.787138    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q"
Nov 03 19:57:17.824816 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.824779    5591 generic.go:356] "Generic (PLEG): container finished" podID="454eb87e-3359-46e5-86e3-36ea2e6f0663" containerID="de8eebc5c083a01ec9db3b8248706f4f1440c593a3fc9362b65fe076c022badf" exitCode=0
Nov 03 19:57:17.825165 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.825140    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t" event={"ID":"454eb87e-3359-46e5-86e3-36ea2e6f0663","Type":"ContainerDied","Data":"de8eebc5c083a01ec9db3b8248706f4f1440c593a3fc9362b65fe076c022badf"}
Nov 03 19:57:17.825570 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.825549    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t" event={"ID":"454eb87e-3359-46e5-86e3-36ea2e6f0663","Type":"ContainerDied","Data":"863d60453aa393518c5903b8e0fcb4b63042ea0f92d0d188ccb07b74c31c088b"}
Nov 03 19:57:17.825787 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.825770    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t"
Nov 03 19:57:17.827904 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.827877    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb" event={"ID":"3ce9e626-faa5-4b01-8849-971bc8def06c","Type":"ContainerStarted","Data":"58892f7915d1785b67a58285bc71c2aa057c2cfc41bf363e4f47c23f18f863ec"}
Nov 03 19:57:17.829460 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.829442    5591 scope.go:122] "RemoveContainer" containerID="a125616aae52771aaf2cda257d4dd54e7a8a055a328683cd11c4b695708b4ab0"
Nov 03 19:57:17.829952 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:57:17.829930    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a125616aae52771aaf2cda257d4dd54e7a8a055a328683cd11c4b695708b4ab0\": container with ID starting with a125616aae52771aaf2cda257d4dd54e7a8a055a328683cd11c4b695708b4ab0 not found: ID does not exist" containerID="a125616aae52771aaf2cda257d4dd54e7a8a055a328683cd11c4b695708b4ab0"
Nov 03 19:57:17.830096 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.830073    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a125616aae52771aaf2cda257d4dd54e7a8a055a328683cd11c4b695708b4ab0"} err="failed to get container status \"a125616aae52771aaf2cda257d4dd54e7a8a055a328683cd11c4b695708b4ab0\": rpc error: code = NotFound desc = could not find container \"a125616aae52771aaf2cda257d4dd54e7a8a055a328683cd11c4b695708b4ab0\": container with ID starting with a125616aae52771aaf2cda257d4dd54e7a8a055a328683cd11c4b695708b4ab0 not found: ID does not exist"
Nov 03 19:57:17.831898 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.830192    5591 scope.go:122] "RemoveContainer" containerID="de8eebc5c083a01ec9db3b8248706f4f1440c593a3fc9362b65fe076c022badf"
Nov 03 19:57:17.844879 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.844851    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q"]
Nov 03 19:57:17.860779 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.860742    5591 scope.go:122] "RemoveContainer" containerID="de8eebc5c083a01ec9db3b8248706f4f1440c593a3fc9362b65fe076c022badf"
Nov 03 19:57:17.861924 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:57:17.861898    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de8eebc5c083a01ec9db3b8248706f4f1440c593a3fc9362b65fe076c022badf\": container with ID starting with de8eebc5c083a01ec9db3b8248706f4f1440c593a3fc9362b65fe076c022badf not found: ID does not exist" containerID="de8eebc5c083a01ec9db3b8248706f4f1440c593a3fc9362b65fe076c022badf"
Nov 03 19:57:17.862075 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.862052    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de8eebc5c083a01ec9db3b8248706f4f1440c593a3fc9362b65fe076c022badf"} err="failed to get container status \"de8eebc5c083a01ec9db3b8248706f4f1440c593a3fc9362b65fe076c022badf\": rpc error: code = NotFound desc = could not find container \"de8eebc5c083a01ec9db3b8248706f4f1440c593a3fc9362b65fe076c022badf\": container with ID starting with de8eebc5c083a01ec9db3b8248706f4f1440c593a3fc9362b65fe076c022badf not found: ID does not exist"
Nov 03 19:57:17.864147 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.864129    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3d1db1-96d59b748-wcl9q"]
Nov 03 19:57:17.877531 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.877315    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/454eb87e-3359-46e5-86e3-36ea2e6f0663-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:17.877531 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.877348    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h652l\" (UniqueName: \"kubernetes.io/projected/454eb87e-3359-46e5-86e3-36ea2e6f0663-kube-api-access-h652l\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:17.895228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.891988    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t"]
Nov 03 19:57:17.896544 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:17.896387    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-59b8fdc4ff-rt24t"]
Nov 03 19:57:18.068418 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:18.068329    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj" podUID="685e1e6c-bc01-40af-beeb-8bbc2f218de5" containerName="payload-processing" containerID="cri-o://825a9526c87e44117f13cf7fa6b898b07d23a8796d03f1b33376033f9c53b37e" gracePeriod=25
Nov 03 19:57:18.445354 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:18.445326    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr"
Nov 03 19:57:18.603441 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:18.603346    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj"
Nov 03 19:57:18.615047 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:18.615001    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/685e1e6c-bc01-40af-beeb-8bbc2f218de5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/685e1e6c-bc01-40af-beeb-8bbc2f218de5-plugins-config-volume\") pod \"685e1e6c-bc01-40af-beeb-8bbc2f218de5\" (UID: \"685e1e6c-bc01-40af-beeb-8bbc2f218de5\") "
Nov 03 19:57:18.615231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:18.615065    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/685e1e6c-bc01-40af-beeb-8bbc2f218de5-kube-api-access-sblhz\" (UniqueName: \"kubernetes.io/projected/685e1e6c-bc01-40af-beeb-8bbc2f218de5-kube-api-access-sblhz\") pod \"685e1e6c-bc01-40af-beeb-8bbc2f218de5\" (UID: \"685e1e6c-bc01-40af-beeb-8bbc2f218de5\") "
Nov 03 19:57:18.615455 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:18.615432    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/685e1e6c-bc01-40af-beeb-8bbc2f218de5-plugins-config-volume" pod "685e1e6c-bc01-40af-beeb-8bbc2f218de5" (UID: "685e1e6c-bc01-40af-beeb-8bbc2f218de5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:57:18.615605 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:18.615590    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/685e1e6c-bc01-40af-beeb-8bbc2f218de5-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:18.617535 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:18.617511    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/685e1e6c-bc01-40af-beeb-8bbc2f218de5-kube-api-access-sblhz" pod "685e1e6c-bc01-40af-beeb-8bbc2f218de5" (UID: "685e1e6c-bc01-40af-beeb-8bbc2f218de5"). InnerVolumeSpecName "kube-api-access-sblhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:57:18.716933 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:18.716896    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sblhz\" (UniqueName: \"kubernetes.io/projected/685e1e6c-bc01-40af-beeb-8bbc2f218de5-kube-api-access-sblhz\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:18.837460 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:18.837404    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb" event={"ID":"3ce9e626-faa5-4b01-8849-971bc8def06c","Type":"ContainerStarted","Data":"d3dce1d5133bd8f4520c79a3c89850fe761a89ee66847e92663879131b39c5c5"}
Nov 03 19:57:18.837648 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:18.837495    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb"
Nov 03 19:57:18.838984 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:18.838952    5591 generic.go:356] "Generic (PLEG): container finished" podID="685e1e6c-bc01-40af-beeb-8bbc2f218de5" containerID="825a9526c87e44117f13cf7fa6b898b07d23a8796d03f1b33376033f9c53b37e" exitCode=0
Nov 03 19:57:18.839140 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:18.838999    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj"
Nov 03 19:57:18.839140 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:18.839038    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj" event={"ID":"685e1e6c-bc01-40af-beeb-8bbc2f218de5","Type":"ContainerDied","Data":"825a9526c87e44117f13cf7fa6b898b07d23a8796d03f1b33376033f9c53b37e"}
Nov 03 19:57:18.839140 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:18.839085    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj" event={"ID":"685e1e6c-bc01-40af-beeb-8bbc2f218de5","Type":"ContainerDied","Data":"eb3b5524be0e2a03ecd0a414843d173cbdc214a1c125f73fc784e9af5f02427b"}
Nov 03 19:57:18.839140 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:18.839100    5591 scope.go:122] "RemoveContainer" containerID="825a9526c87e44117f13cf7fa6b898b07d23a8796d03f1b33376033f9c53b37e"
Nov 03 19:57:18.851787 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:18.851767    5591 scope.go:122] "RemoveContainer" containerID="825a9526c87e44117f13cf7fa6b898b07d23a8796d03f1b33376033f9c53b37e"
Nov 03 19:57:18.852141 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:57:18.852121    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"825a9526c87e44117f13cf7fa6b898b07d23a8796d03f1b33376033f9c53b37e\": container with ID starting with 825a9526c87e44117f13cf7fa6b898b07d23a8796d03f1b33376033f9c53b37e not found: ID does not exist" containerID="825a9526c87e44117f13cf7fa6b898b07d23a8796d03f1b33376033f9c53b37e"
Nov 03 19:57:18.852234 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:18.852152    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"825a9526c87e44117f13cf7fa6b898b07d23a8796d03f1b33376033f9c53b37e"} err="failed to get container status \"825a9526c87e44117f13cf7fa6b898b07d23a8796d03f1b33376033f9c53b37e\": rpc error: code = NotFound desc = could not find container \"825a9526c87e44117f13cf7fa6b898b07d23a8796d03f1b33376033f9c53b37e\": container with ID starting with 825a9526c87e44117f13cf7fa6b898b07d23a8796d03f1b33376033f9c53b37e not found: ID does not exist"
Nov 03 19:57:18.871768 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:18.871668    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb" podStartSLOduration=2.41530284 podStartE2EDuration="2.871652594s" totalImagesPullingTime="456.349754ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:18.869603946 +0000 UTC m=+1187.641896964" watchObservedRunningTime="2026-11-03 19:57:18.871652594 +0000 UTC m=+1187.643945613"
Nov 03 19:57:18.888540 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:18.888505    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj"]
Nov 03 19:57:18.893373 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:18.893342    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9b3619-749bcdcf58-4zzgj"]
Nov 03 19:57:19.352886 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:19.352850    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="454eb87e-3359-46e5-86e3-36ea2e6f0663" path="/var/lib/kubelet/pods/454eb87e-3359-46e5-86e3-36ea2e6f0663/volumes"
Nov 03 19:57:19.353541 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:19.353518    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="685e1e6c-bc01-40af-beeb-8bbc2f218de5" path="/var/lib/kubelet/pods/685e1e6c-bc01-40af-beeb-8bbc2f218de5/volumes"
Nov 03 19:57:19.353967 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:19.353947    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8618c3c2-38f6-45fd-9e7b-9be8db40da4e" path="/var/lib/kubelet/pods/8618c3c2-38f6-45fd-9e7b-9be8db40da4e/volumes"
Nov 03 19:57:22.042539 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.042500    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5"]
Nov 03 19:57:22.043584 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.043559    5591 cpu_manager.go:414] "containerMap: removing container" podUID="8618c3c2-38f6-45fd-9e7b-9be8db40da4e" containerName="payload-processing"
Nov 03 19:57:22.043584 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.043585    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8618c3c2-38f6-45fd-9e7b-9be8db40da4e" containerName="payload-processing"
Nov 03 19:57:22.043794 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.043626    5591 cpu_manager.go:414] "containerMap: removing container" podUID="685e1e6c-bc01-40af-beeb-8bbc2f218de5" containerName="payload-processing"
Nov 03 19:57:22.043794 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.043633    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="685e1e6c-bc01-40af-beeb-8bbc2f218de5" containerName="payload-processing"
Nov 03 19:57:22.043794 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.043673    5591 cpu_manager.go:414] "containerMap: removing container" podUID="454eb87e-3359-46e5-86e3-36ea2e6f0663" containerName="payload-pre-processing"
Nov 03 19:57:22.043794 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.043683    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="454eb87e-3359-46e5-86e3-36ea2e6f0663" containerName="payload-pre-processing"
Nov 03 19:57:22.044006 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.043875    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="454eb87e-3359-46e5-86e3-36ea2e6f0663" containerName="payload-pre-processing"
Nov 03 19:57:22.044006 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.043902    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8618c3c2-38f6-45fd-9e7b-9be8db40da4e" containerName="payload-processing"
Nov 03 19:57:22.044006 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.043916    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="685e1e6c-bc01-40af-beeb-8bbc2f218de5" containerName="payload-processing"
Nov 03 19:57:22.050444 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.050418    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5"
Nov 03 19:57:22.056095 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.056072    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5"]
Nov 03 19:57:22.104243 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.101605    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dqjj\" (UniqueName: \"kubernetes.io/projected/09c9289a-f596-4c46-b4a0-a85cc1cc70e5-kube-api-access-9dqjj\") pod \"payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5\" (UID: \"09c9289a-f596-4c46-b4a0-a85cc1cc70e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5"
Nov 03 19:57:22.104243 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.101742    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09c9289a-f596-4c46-b4a0-a85cc1cc70e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5\" (UID: \"09c9289a-f596-4c46-b4a0-a85cc1cc70e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5"
Nov 03 19:57:22.133567 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.133531    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3d1db1-dockercfg-9m98k\""
Nov 03 19:57:22.170285 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.133927    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3d1db1\""
Nov 03 19:57:22.228231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.212382    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09c9289a-f596-4c46-b4a0-a85cc1cc70e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5\" (UID: \"09c9289a-f596-4c46-b4a0-a85cc1cc70e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5"
Nov 03 19:57:22.228231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.212646    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dqjj\" (UniqueName: \"kubernetes.io/projected/09c9289a-f596-4c46-b4a0-a85cc1cc70e5-kube-api-access-9dqjj\") pod \"payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5\" (UID: \"09c9289a-f596-4c46-b4a0-a85cc1cc70e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5"
Nov 03 19:57:22.228231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.213852    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09c9289a-f596-4c46-b4a0-a85cc1cc70e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5\" (UID: \"09c9289a-f596-4c46-b4a0-a85cc1cc70e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5"
Nov 03 19:57:22.432646 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.430747    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb"]
Nov 03 19:57:22.432646 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.430835    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"]
Nov 03 19:57:22.439879 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.439838    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"]
Nov 03 19:57:22.440048 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.439951    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.451756 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.451699    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dqjj\" (UniqueName: \"kubernetes.io/projected/09c9289a-f596-4c46-b4a0-a85cc1cc70e5-kube-api-access-9dqjj\") pod \"payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5\" (UID: \"09c9289a-f596-4c46-b4a0-a85cc1cc70e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5"
Nov 03 19:57:22.466619 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.461740    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-3d1db1-openshift-default-dockercfg-2ndm2\""
Nov 03 19:57:22.479891 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.479856    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66"]
Nov 03 19:57:22.480148 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.480119    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66" podUID="7cca4af3-235a-4f69-8cad-a752fea612f6" containerName="maas-api" containerID="cri-o://53c82f35ecb9fca90c73db144d85c0e8fe56cd0fe464cc49e511c4ffc77db0f2" gracePeriod=30
Nov 03 19:57:22.537384 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.537346    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-podinfo\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.537541 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.537397    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-data\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.537541 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.537455    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-token\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.537541 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.537491    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-envoy\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.537727 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.537549    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-workload-certs\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.537727 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.537588    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w85xr\" (UniqueName: \"kubernetes.io/projected/bec03382-01ff-4d4d-89a6-2988ddb9e188-kube-api-access-w85xr\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.537831 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.537737    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-workload-socket\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.537897 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.537864    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec03382-01ff-4d4d-89a6-2988ddb9e188-istiod-ca-cert\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.537946 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.537910    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-credential-socket\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.639468 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.639418    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-podinfo\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.639468 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.639466    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-data\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.639728 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.639605    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-token\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.639728 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.639655    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-envoy\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.639728 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.639724    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-workload-certs\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.639880 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.639767    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w85xr\" (UniqueName: \"kubernetes.io/projected/bec03382-01ff-4d4d-89a6-2988ddb9e188-kube-api-access-w85xr\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.639931 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.639893    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-data\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.639986 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.639929    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-workload-socket\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.640078 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.640047    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec03382-01ff-4d4d-89a6-2988ddb9e188-istiod-ca-cert\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.640136 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.640108    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-credential-socket\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.640187 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.640158    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-workload-certs\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.640777 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.640745    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-credential-socket\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.640875 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.640792    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-workload-socket\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.641097 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.641073    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec03382-01ff-4d4d-89a6-2988ddb9e188-istiod-ca-cert\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.642447 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.642423    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-envoy\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.642556 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.642501    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-podinfo\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.652348 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.652320    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-token\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.652489 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.652419    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w85xr\" (UniqueName: \"kubernetes.io/projected/bec03382-01ff-4d4d-89a6-2988ddb9e188-kube-api-access-w85xr\") pod \"e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") " pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.673195 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.673161    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5"
Nov 03 19:57:22.813625 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.811872    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:22.957336 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.949651    5591 generic.go:356] "Generic (PLEG): container finished" podID="7cca4af3-235a-4f69-8cad-a752fea612f6" containerID="53c82f35ecb9fca90c73db144d85c0e8fe56cd0fe464cc49e511c4ffc77db0f2" exitCode=0
Nov 03 19:57:22.957336 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:22.949781    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66" event={"ID":"7cca4af3-235a-4f69-8cad-a752fea612f6","Type":"ContainerDied","Data":"53c82f35ecb9fca90c73db144d85c0e8fe56cd0fe464cc49e511c4ffc77db0f2"}
Nov 03 19:57:23.095485 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:23.092126    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66"
Nov 03 19:57:23.141941 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:23.141895    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cca4af3-235a-4f69-8cad-a752fea612f6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cca4af3-235a-4f69-8cad-a752fea612f6-metrics-certs\") pod \"7cca4af3-235a-4f69-8cad-a752fea612f6\" (UID: \"7cca4af3-235a-4f69-8cad-a752fea612f6\") "
Nov 03 19:57:23.142122 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:23.141943    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cca4af3-235a-4f69-8cad-a752fea612f6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cca4af3-235a-4f69-8cad-a752fea612f6-maas-api-tls\") pod \"7cca4af3-235a-4f69-8cad-a752fea612f6\" (UID: \"7cca4af3-235a-4f69-8cad-a752fea612f6\") "
Nov 03 19:57:23.147890 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:23.147856    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cca4af3-235a-4f69-8cad-a752fea612f6-metrics-certs" pod "7cca4af3-235a-4f69-8cad-a752fea612f6" (UID: "7cca4af3-235a-4f69-8cad-a752fea612f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:57:23.151310 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:23.151263    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cca4af3-235a-4f69-8cad-a752fea612f6-maas-api-tls" pod "7cca4af3-235a-4f69-8cad-a752fea612f6" (UID: "7cca4af3-235a-4f69-8cad-a752fea612f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:57:23.243181 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:23.243073    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cca4af3-235a-4f69-8cad-a752fea612f6-kube-api-access-2ztxd\" (UniqueName: \"kubernetes.io/projected/7cca4af3-235a-4f69-8cad-a752fea612f6-kube-api-access-2ztxd\") pod \"7cca4af3-235a-4f69-8cad-a752fea612f6\" (UID: \"7cca4af3-235a-4f69-8cad-a752fea612f6\") "
Nov 03 19:57:23.243907 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:23.243504    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cca4af3-235a-4f69-8cad-a752fea612f6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cca4af3-235a-4f69-8cad-a752fea612f6-openshift-service-ca\") pod \"7cca4af3-235a-4f69-8cad-a752fea612f6\" (UID: \"7cca4af3-235a-4f69-8cad-a752fea612f6\") "
Nov 03 19:57:23.244451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:23.244428    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cca4af3-235a-4f69-8cad-a752fea612f6-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:23.244596 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:23.244455    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cca4af3-235a-4f69-8cad-a752fea612f6-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:23.244833 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:23.244809    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cca4af3-235a-4f69-8cad-a752fea612f6-openshift-service-ca" pod "7cca4af3-235a-4f69-8cad-a752fea612f6" (UID: "7cca4af3-235a-4f69-8cad-a752fea612f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:57:23.247278 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:23.247248    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cca4af3-235a-4f69-8cad-a752fea612f6-kube-api-access-2ztxd" pod "7cca4af3-235a-4f69-8cad-a752fea612f6" (UID: "7cca4af3-235a-4f69-8cad-a752fea612f6"). InnerVolumeSpecName "kube-api-access-2ztxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:57:23.369855 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:23.350750    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ztxd\" (UniqueName: \"kubernetes.io/projected/7cca4af3-235a-4f69-8cad-a752fea612f6-kube-api-access-2ztxd\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:23.369855 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:23.350791    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cca4af3-235a-4f69-8cad-a752fea612f6-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:23.414956 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:57:23.414921    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbec03382_01ff_4d4d_89a6_2988ddb9e188.slice/crio-93f9eb320edd1f05c4cf9a48618730ea655342f5ee7c57c8cab0118371284789 WatchSource:0}: Error finding container 93f9eb320edd1f05c4cf9a48618730ea655342f5ee7c57c8cab0118371284789: Status 404 returned error can't find the container with id 93f9eb320edd1f05c4cf9a48618730ea655342f5ee7c57c8cab0118371284789
Nov 03 19:57:23.415296 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:23.415276    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5"]
Nov 03 19:57:23.418528 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:23.418497    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"]
Nov 03 19:57:23.418831 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:23.418788    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:57:23.418943 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:23.418881    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:57:23.418943 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:23.418930    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:57:23.969272 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:23.965680    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5" event={"ID":"09c9289a-f596-4c46-b4a0-a85cc1cc70e5","Type":"ContainerStarted","Data":"06918d60b14722d2e9623471452e39df3f73eab21330d78370311567922c082a"}
Nov 03 19:57:23.969272 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:23.967585    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj" event={"ID":"bec03382-01ff-4d4d-89a6-2988ddb9e188","Type":"ContainerStarted","Data":"7364cc3f424c4fee6e9f69edccc249f0788728588bc80f11f82f307656b00bf1"}
Nov 03 19:57:23.969272 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:23.967632    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj" event={"ID":"bec03382-01ff-4d4d-89a6-2988ddb9e188","Type":"ContainerStarted","Data":"93f9eb320edd1f05c4cf9a48618730ea655342f5ee7c57c8cab0118371284789"}
Nov 03 19:57:23.970310 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:23.970260    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66" event={"ID":"7cca4af3-235a-4f69-8cad-a752fea612f6","Type":"ContainerDied","Data":"1176eed55abb066d28fe078bac8d5c9571bec58da848f8d204f306fd3b38bafd"}
Nov 03 19:57:23.970357 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:23.970326    5591 scope.go:122] "RemoveContainer" containerID="53c82f35ecb9fca90c73db144d85c0e8fe56cd0fe464cc49e511c4ffc77db0f2"
Nov 03 19:57:23.970500 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:23.970486    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66"
Nov 03 19:57:23.996384 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:23.996328    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj" podStartSLOduration=1.9963118450000001 podStartE2EDuration="1.996311845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:23.993564734 +0000 UTC m=+1192.765857775" watchObservedRunningTime="2026-11-03 19:57:23.996311845 +0000 UTC m=+1192.768604863"
Nov 03 19:57:24.022530 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:24.022491    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66"]
Nov 03 19:57:24.027747 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:24.027719    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66"]
Nov 03 19:57:24.323745 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:24.319831    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7"]
Nov 03 19:57:24.323745 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:24.320589    5591 cpu_manager.go:414] "containerMap: removing container" podUID="7cca4af3-235a-4f69-8cad-a752fea612f6" containerName="maas-api"
Nov 03 19:57:24.323745 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:24.320605    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cca4af3-235a-4f69-8cad-a752fea612f6" containerName="maas-api"
Nov 03 19:57:24.323745 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:24.320720    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cca4af3-235a-4f69-8cad-a752fea612f6" containerName="maas-api"
Nov 03 19:57:24.329744 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:24.329715    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7"
Nov 03 19:57:24.337650 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:24.337625    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7"]
Nov 03 19:57:24.344424 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:24.340552    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9b3619\""
Nov 03 19:57:24.415189 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:24.415143    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vntqm\" (UniqueName: \"kubernetes.io/projected/0d82d43e-048c-4e90-968b-a86c91ca38a2-kube-api-access-vntqm\") pod \"payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7\" (UID: \"0d82d43e-048c-4e90-968b-a86c91ca38a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7"
Nov 03 19:57:24.415533 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:24.415503    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d82d43e-048c-4e90-968b-a86c91ca38a2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7\" (UID: \"0d82d43e-048c-4e90-968b-a86c91ca38a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7"
Nov 03 19:57:24.535227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:24.521179    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d82d43e-048c-4e90-968b-a86c91ca38a2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7\" (UID: \"0d82d43e-048c-4e90-968b-a86c91ca38a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7"
Nov 03 19:57:24.561271 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:24.557761    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vntqm\" (UniqueName: \"kubernetes.io/projected/0d82d43e-048c-4e90-968b-a86c91ca38a2-kube-api-access-vntqm\") pod \"payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7\" (UID: \"0d82d43e-048c-4e90-968b-a86c91ca38a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7"
Nov 03 19:57:24.561271 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:24.536359    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d82d43e-048c-4e90-968b-a86c91ca38a2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7\" (UID: \"0d82d43e-048c-4e90-968b-a86c91ca38a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7"
Nov 03 19:57:24.689278 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:24.688068    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vntqm\" (UniqueName: \"kubernetes.io/projected/0d82d43e-048c-4e90-968b-a86c91ca38a2-kube-api-access-vntqm\") pod \"payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7\" (UID: \"0d82d43e-048c-4e90-968b-a86c91ca38a2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7"
Nov 03 19:57:24.697229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:24.693947    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7"
Nov 03 19:57:24.836572 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:24.836531    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:24.838377 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:24.838347    5591 patch_prober.go:29] interesting pod/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 03 19:57:24.838539 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:24.838402    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj" podUID="bec03382-01ff-4d4d-89a6-2988ddb9e188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 03 19:57:24.862244 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:24.861089    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb"
Nov 03 19:57:25.357144 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:25.357111    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cca4af3-235a-4f69-8cad-a752fea612f6" path="/var/lib/kubelet/pods/7cca4af3-235a-4f69-8cad-a752fea612f6/volumes"
Nov 03 19:57:25.446504 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:25.441797    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7"]
Nov 03 19:57:25.822454 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:25.822412    5591 patch_prober.go:29] interesting pod/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 03 19:57:25.822633 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:25.822527    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj" podUID="bec03382-01ff-4d4d-89a6-2988ddb9e188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 03 19:57:26.027231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:26.023373    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b0f9d6-6df55bb7d8-4nq66" podUID="7cca4af3-235a-4f69-8cad-a752fea612f6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.110:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:57:26.247305 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:26.235347    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7" event={"ID":"0d82d43e-048c-4e90-968b-a86c91ca38a2","Type":"ContainerStarted","Data":"106f63835eaee38a7fb3c22e72f020c5847a0721f19f0da2e9c35ff74ae898d6"}
Nov 03 19:57:26.821804 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:26.821764    5591 patch_prober.go:29] interesting pod/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 03 19:57:26.822340 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:26.821831    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj" podUID="bec03382-01ff-4d4d-89a6-2988ddb9e188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 03 19:57:27.271076 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.271043    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"]
Nov 03 19:57:27.320252 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.318805    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"]
Nov 03 19:57:27.323635 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.323082    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.447349 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.444805    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb" podUID="6a5c139a-00fc-40e3-8917-df4391456425" containerName="payload-processing" containerID="cri-o://d647cc15f8b68add527a3a033b5fcf8bbaac01ab3b12ea84ad8e1a24772d1538" gracePeriod=25
Nov 03 19:57:27.465904 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.451526    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3b6e2b-openshift-default-dockercfg-7r78b\""
Nov 03 19:57:27.589509 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.554400    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-workload-certs\") pod \"e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") " pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.589509 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.554454    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-workload-socket\") pod \"e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") " pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.589509 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.554602    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-istio-data\") pod \"e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") " pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.589509 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.554660    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-istio-envoy\") pod \"e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") " pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.589509 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.554696    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zjcw\" (UniqueName: \"kubernetes.io/projected/f226a9f7-2809-4890-8cc3-29013e49d774-kube-api-access-5zjcw\") pod \"e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") " pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.589509 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.554815    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f226a9f7-2809-4890-8cc3-29013e49d774-istiod-ca-cert\") pod \"e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") " pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.589509 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.554872    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f226a9f7-2809-4890-8cc3-29013e49d774-istio-token\") pod \"e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") " pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.589509 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.554909    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-credential-socket\") pod \"e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") " pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.589509 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.554983    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f226a9f7-2809-4890-8cc3-29013e49d774-istio-podinfo\") pod \"e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") " pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.673790 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.656152    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-workload-certs\") pod \"e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") " pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.673790 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.656229    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-workload-socket\") pod \"e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") " pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.673790 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.656354    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-istio-data\") pod \"e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") " pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.673790 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.656406    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-istio-envoy\") pod \"e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") " pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.673790 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.656439    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zjcw\" (UniqueName: \"kubernetes.io/projected/f226a9f7-2809-4890-8cc3-29013e49d774-kube-api-access-5zjcw\") pod \"e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") " pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.673790 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.656488    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f226a9f7-2809-4890-8cc3-29013e49d774-istiod-ca-cert\") pod \"e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") " pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.673790 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.656541    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f226a9f7-2809-4890-8cc3-29013e49d774-istio-token\") pod \"e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") " pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.673790 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.656575    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-credential-socket\") pod \"e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") " pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.673790 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.656648    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f226a9f7-2809-4890-8cc3-29013e49d774-istio-podinfo\") pod \"e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") " pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.674451 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.673911    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f226a9f7-2809-4890-8cc3-29013e49d774-istiod-ca-cert\") pod \"e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") " pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.677709 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.676735    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-workload-certs\") pod \"e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") " pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.678391 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.678349    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-credential-socket\") pod \"e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") " pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.678686 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.678657    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-workload-socket\") pod \"e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") " pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.679231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.679186    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-istio-data\") pod \"e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") " pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.727886 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.727807    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f226a9f7-2809-4890-8cc3-29013e49d774-istio-token\") pod \"e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") " pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.812541 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.799395    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f226a9f7-2809-4890-8cc3-29013e49d774-istio-podinfo\") pod \"e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") " pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.824634 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.823286    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zjcw\" (UniqueName: \"kubernetes.io/projected/f226a9f7-2809-4890-8cc3-29013e49d774-kube-api-access-5zjcw\") pod \"e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") " pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.824634 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.823783    5591 patch_prober.go:29] interesting pod/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 03 19:57:27.824634 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.823862    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj" podUID="bec03382-01ff-4d4d-89a6-2988ddb9e188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 03 19:57:27.825073 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.824837    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-istio-envoy\") pod \"e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") " pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:27.948561 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:27.948306    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:28.055592 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:28.055230    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb"
Nov 03 19:57:28.173018 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:28.172972    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a5c139a-00fc-40e3-8917-df4391456425-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5c139a-00fc-40e3-8917-df4391456425-plugins-config-volume\") pod \"6a5c139a-00fc-40e3-8917-df4391456425\" (UID: \"6a5c139a-00fc-40e3-8917-df4391456425\") "
Nov 03 19:57:28.173276 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:28.173195    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a5c139a-00fc-40e3-8917-df4391456425-kube-api-access-h488f\" (UniqueName: \"kubernetes.io/projected/6a5c139a-00fc-40e3-8917-df4391456425-kube-api-access-h488f\") pod \"6a5c139a-00fc-40e3-8917-df4391456425\" (UID: \"6a5c139a-00fc-40e3-8917-df4391456425\") "
Nov 03 19:57:28.174494 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:28.174464    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5c139a-00fc-40e3-8917-df4391456425-plugins-config-volume" pod "6a5c139a-00fc-40e3-8917-df4391456425" (UID: "6a5c139a-00fc-40e3-8917-df4391456425"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:57:28.202432 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:28.200889    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5c139a-00fc-40e3-8917-df4391456425-kube-api-access-h488f" pod "6a5c139a-00fc-40e3-8917-df4391456425" (UID: "6a5c139a-00fc-40e3-8917-df4391456425"). InnerVolumeSpecName "kube-api-access-h488f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:57:28.270718 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:28.270608    5591 generic.go:356] "Generic (PLEG): container finished" podID="6a5c139a-00fc-40e3-8917-df4391456425" containerID="d647cc15f8b68add527a3a033b5fcf8bbaac01ab3b12ea84ad8e1a24772d1538" exitCode=0
Nov 03 19:57:28.270873 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:28.270792    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb" event={"ID":"6a5c139a-00fc-40e3-8917-df4391456425","Type":"ContainerDied","Data":"d647cc15f8b68add527a3a033b5fcf8bbaac01ab3b12ea84ad8e1a24772d1538"}
Nov 03 19:57:28.270873 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:28.270848    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb" event={"ID":"6a5c139a-00fc-40e3-8917-df4391456425","Type":"ContainerDied","Data":"808d9863346d4b06a2f564d48be72f69ed70117b6253eb3ea2ce32af41062156"}
Nov 03 19:57:28.271007 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:28.270876    5591 scope.go:122] "RemoveContainer" containerID="d647cc15f8b68add527a3a033b5fcf8bbaac01ab3b12ea84ad8e1a24772d1538"
Nov 03 19:57:28.271058 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:28.271033    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb"
Nov 03 19:57:28.275088 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:28.274926    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h488f\" (UniqueName: \"kubernetes.io/projected/6a5c139a-00fc-40e3-8917-df4391456425-kube-api-access-h488f\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:28.275088 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:28.274950    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5c139a-00fc-40e3-8917-df4391456425-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:28.401229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:28.400803    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb"]
Nov 03 19:57:28.437154 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:28.436740    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b34f6c-78755bfcb6-q4txb"]
Nov 03 19:57:28.455930 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:28.455909    5591 scope.go:122] "RemoveContainer" containerID="d647cc15f8b68add527a3a033b5fcf8bbaac01ab3b12ea84ad8e1a24772d1538"
Nov 03 19:57:28.456357 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:57:28.456332    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d647cc15f8b68add527a3a033b5fcf8bbaac01ab3b12ea84ad8e1a24772d1538\": container with ID starting with d647cc15f8b68add527a3a033b5fcf8bbaac01ab3b12ea84ad8e1a24772d1538 not found: ID does not exist" containerID="d647cc15f8b68add527a3a033b5fcf8bbaac01ab3b12ea84ad8e1a24772d1538"
Nov 03 19:57:28.456465 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:28.456371    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d647cc15f8b68add527a3a033b5fcf8bbaac01ab3b12ea84ad8e1a24772d1538"} err="failed to get container status \"d647cc15f8b68add527a3a033b5fcf8bbaac01ab3b12ea84ad8e1a24772d1538\": rpc error: code = NotFound desc = could not find container \"d647cc15f8b68add527a3a033b5fcf8bbaac01ab3b12ea84ad8e1a24772d1538\": container with ID starting with d647cc15f8b68add527a3a033b5fcf8bbaac01ab3b12ea84ad8e1a24772d1538 not found: ID does not exist"
Nov 03 19:57:28.566297 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:57:28.566196    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf226a9f7_2809_4890_8cc3_29013e49d774.slice/crio-27ca9f5c4895cfcf55aed14435bbcb9763950cb21ff1e5cbb2d52536e0cf7885 WatchSource:0}: Error finding container 27ca9f5c4895cfcf55aed14435bbcb9763950cb21ff1e5cbb2d52536e0cf7885: Status 404 returned error can't find the container with id 27ca9f5c4895cfcf55aed14435bbcb9763950cb21ff1e5cbb2d52536e0cf7885
Nov 03 19:57:28.566620 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:28.566599    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"]
Nov 03 19:57:28.585235 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:28.576705    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:57:28.585235 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:28.576826    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:57:28.585235 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:28.576873    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:57:28.748167 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:57:28.747681    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:57:28.823766 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:28.823727    5591 patch_prober.go:29] interesting pod/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 03 19:57:28.823995 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:28.823790    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj" podUID="bec03382-01ff-4d4d-89a6-2988ddb9e188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 03 19:57:29.025256 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:29.025185    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf"]
Nov 03 19:57:29.025851 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:29.025817    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf" podUID="3f2c38b0-cb8a-4fec-9630-be624f51ff9d" containerName="maas-api" containerID="cri-o://a64756151782805657df1e21a12f61c11b34a9ac0c06643066c93197a7da9d14" gracePeriod=30
Nov 03 19:57:29.832639 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:29.829752    5591 patch_prober.go:29] interesting pod/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 03 19:57:29.832639 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:29.829844    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj" podUID="bec03382-01ff-4d4d-89a6-2988ddb9e188" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 03 19:57:29.938088 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:29.935579    5591 generic.go:356] "Generic (PLEG): container finished" podID="3f2c38b0-cb8a-4fec-9630-be624f51ff9d" containerID="a64756151782805657df1e21a12f61c11b34a9ac0c06643066c93197a7da9d14" exitCode=0
Nov 03 19:57:29.945534 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:29.945503    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a5c139a-00fc-40e3-8917-df4391456425" path="/var/lib/kubelet/pods/6a5c139a-00fc-40e3-8917-df4391456425/volumes"
Nov 03 19:57:29.946139 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:29.946119    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf" event={"ID":"3f2c38b0-cb8a-4fec-9630-be624f51ff9d","Type":"ContainerDied","Data":"a64756151782805657df1e21a12f61c11b34a9ac0c06643066c93197a7da9d14"}
Nov 03 19:57:29.953742 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:29.953714    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf"
Nov 03 19:57:29.955816 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:29.955775    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5" event={"ID":"09c9289a-f596-4c46-b4a0-a85cc1cc70e5","Type":"ContainerStarted","Data":"b10f53fa7418fd888b875d696e0f2f0bd04e2a81946b57498b329d2d513b1229"}
Nov 03 19:57:29.956864 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:29.956833    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5"
Nov 03 19:57:29.960067 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:29.960039    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq" event={"ID":"f226a9f7-2809-4890-8cc3-29013e49d774","Type":"ContainerStarted","Data":"8842dc2ff9cb2bc31ba58bbcedbd7f0d914cd991a91fc3b5c2352b06cdb48090"}
Nov 03 19:57:29.960176 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:29.960084    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq" event={"ID":"f226a9f7-2809-4890-8cc3-29013e49d774","Type":"ContainerStarted","Data":"27ca9f5c4895cfcf55aed14435bbcb9763950cb21ff1e5cbb2d52536e0cf7885"}
Nov 03 19:57:30.018281 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:30.018186    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5" podStartSLOduration=2.3933963560000002 podStartE2EDuration="8.018165044s" totalImagesPullingTime="5.624768688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:30.007739177 +0000 UTC m=+1198.780032196" watchObservedRunningTime="2026-11-03 19:57:30.018165044 +0000 UTC m=+1198.790458063"
Nov 03 19:57:30.071055 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:30.071021    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-openshift-service-ca\") pod \"3f2c38b0-cb8a-4fec-9630-be624f51ff9d\" (UID: \"3f2c38b0-cb8a-4fec-9630-be624f51ff9d\") "
Nov 03 19:57:30.071541 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:30.071103    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-maas-api-tls\") pod \"3f2c38b0-cb8a-4fec-9630-be624f51ff9d\" (UID: \"3f2c38b0-cb8a-4fec-9630-be624f51ff9d\") "
Nov 03 19:57:30.071541 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:30.071162    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-kube-api-access-9lb54\" (UniqueName: \"kubernetes.io/projected/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-kube-api-access-9lb54\") pod \"3f2c38b0-cb8a-4fec-9630-be624f51ff9d\" (UID: \"3f2c38b0-cb8a-4fec-9630-be624f51ff9d\") "
Nov 03 19:57:30.071541 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:30.071275    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-metrics-certs\") pod \"3f2c38b0-cb8a-4fec-9630-be624f51ff9d\" (UID: \"3f2c38b0-cb8a-4fec-9630-be624f51ff9d\") "
Nov 03 19:57:30.071846 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:30.071820    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-openshift-service-ca" pod "3f2c38b0-cb8a-4fec-9630-be624f51ff9d" (UID: "3f2c38b0-cb8a-4fec-9630-be624f51ff9d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:57:30.072354 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:30.072331    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:30.073916 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:30.073894    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-maas-api-tls" pod "3f2c38b0-cb8a-4fec-9630-be624f51ff9d" (UID: "3f2c38b0-cb8a-4fec-9630-be624f51ff9d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:57:30.074250 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:30.074196    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-metrics-certs" pod "3f2c38b0-cb8a-4fec-9630-be624f51ff9d" (UID: "3f2c38b0-cb8a-4fec-9630-be624f51ff9d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:57:30.074250 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:30.074148    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-kube-api-access-9lb54" pod "3f2c38b0-cb8a-4fec-9630-be624f51ff9d" (UID: "3f2c38b0-cb8a-4fec-9630-be624f51ff9d"). InnerVolumeSpecName "kube-api-access-9lb54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:57:30.192798 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:30.192702    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:30.193039 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:30.193018    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lb54\" (UniqueName: \"kubernetes.io/projected/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-kube-api-access-9lb54\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:30.193139 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:30.193128    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f2c38b0-cb8a-4fec-9630-be624f51ff9d-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:30.826353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:30.826322    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:30.826631 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:30.826610    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:30.827295 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:30.827278    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 19:57:30.860294 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:30.860137    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq" podStartSLOduration=3.860122232 podStartE2EDuration="3.860122232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:30.033665072 +0000 UTC m=+1198.805958091" watchObservedRunningTime="2026-11-03 19:57:30.860122232 +0000 UTC m=+1199.632415249"
Nov 03 19:57:30.948929 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:30.948894    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:30.955176 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:30.955148    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:30.969363 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:30.969333    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf"
Nov 03 19:57:30.969363 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:30.969357    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf" event={"ID":"3f2c38b0-cb8a-4fec-9630-be624f51ff9d","Type":"ContainerDied","Data":"28794cd52864debdc72bdfda495447d01cdce2d8a3714eaeaa3b6c0140c964df"}
Nov 03 19:57:30.969572 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:30.969418    5591 scope.go:122] "RemoveContainer" containerID="a64756151782805657df1e21a12f61c11b34a9ac0c06643066c93197a7da9d14"
Nov 03 19:57:30.971360 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:30.971325    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7" event={"ID":"0d82d43e-048c-4e90-968b-a86c91ca38a2","Type":"ContainerStarted","Data":"e5a9eaf5cdd5dd786235cfd73d0c3409bb186b02a25d575e22a8b0cf1c7ef866"}
Nov 03 19:57:30.971507 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:30.971402    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7"
Nov 03 19:57:30.972079 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:30.971908    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:30.972521 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:30.972506    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:57:30.999425 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:30.999395    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf"]
Nov 03 19:57:31.002563 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:31.002539    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b34f6c-67cc489b9b-hmcgf"]
Nov 03 19:57:31.096340 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:31.078996    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7" podStartSLOduration=2.713709328 podStartE2EDuration="7.078976575s" totalImagesPullingTime="4.365267247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:31.033894646 +0000 UTC m=+1199.806187688" watchObservedRunningTime="2026-11-03 19:57:31.078976575 +0000 UTC m=+1199.851269595"
Nov 03 19:57:31.366410 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:31.364432    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f2c38b0-cb8a-4fec-9630-be624f51ff9d" path="/var/lib/kubelet/pods/3f2c38b0-cb8a-4fec-9630-be624f51ff9d/volumes"
Nov 03 19:57:31.583627 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:31.583596    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 19:57:31.583809 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:31.583601    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 19:57:34.637582 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:34.637544    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b"]
Nov 03 19:57:34.638370 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:34.638353    5591 cpu_manager.go:414] "containerMap: removing container" podUID="6a5c139a-00fc-40e3-8917-df4391456425" containerName="payload-processing"
Nov 03 19:57:34.638444 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:34.638371    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a5c139a-00fc-40e3-8917-df4391456425" containerName="payload-processing"
Nov 03 19:57:34.638444 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:34.638409    5591 cpu_manager.go:414] "containerMap: removing container" podUID="3f2c38b0-cb8a-4fec-9630-be624f51ff9d" containerName="maas-api"
Nov 03 19:57:34.638444 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:34.638414    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f2c38b0-cb8a-4fec-9630-be624f51ff9d" containerName="maas-api"
Nov 03 19:57:34.638541 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:34.638514    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a5c139a-00fc-40e3-8917-df4391456425" containerName="payload-processing"
Nov 03 19:57:34.638541 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:34.638528    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f2c38b0-cb8a-4fec-9630-be624f51ff9d" containerName="maas-api"
Nov 03 19:57:34.644364 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:34.644343    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b"
Nov 03 19:57:34.656314 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:34.656286    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b"]
Nov 03 19:57:34.656657 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:34.656628    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3b6e2b\""
Nov 03 19:57:34.711878 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:34.711844    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z"]
Nov 03 19:57:34.716440 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:34.716420    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z"
Nov 03 19:57:34.729026 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:34.728469    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3b6e2b-dockercfg-4qbk9\""
Nov 03 19:57:34.732494 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:34.732469    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z"]
Nov 03 19:57:34.763411 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:34.763364    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd388f7-86d4-4f57-8438-342a69aff43b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b\" (UID: \"dbd388f7-86d4-4f57-8438-342a69aff43b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b"
Nov 03 19:57:34.818232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:34.796294    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh7vv\" (UniqueName: \"kubernetes.io/projected/dbd388f7-86d4-4f57-8438-342a69aff43b-kube-api-access-xh7vv\") pod \"payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b\" (UID: \"dbd388f7-86d4-4f57-8438-342a69aff43b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b"
Nov 03 19:57:34.917327 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:34.906220    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd388f7-86d4-4f57-8438-342a69aff43b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b\" (UID: \"dbd388f7-86d4-4f57-8438-342a69aff43b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b"
Nov 03 19:57:34.917327 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:34.906302    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh7vv\" (UniqueName: \"kubernetes.io/projected/dbd388f7-86d4-4f57-8438-342a69aff43b-kube-api-access-xh7vv\") pod \"payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b\" (UID: \"dbd388f7-86d4-4f57-8438-342a69aff43b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b"
Nov 03 19:57:34.917327 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:34.906407    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09eb8585-91d8-41a7-919a-77d22fefd3ce-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z\" (UID: \"09eb8585-91d8-41a7-919a-77d22fefd3ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z"
Nov 03 19:57:34.917327 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:34.906458    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t5vn\" (UniqueName: \"kubernetes.io/projected/09eb8585-91d8-41a7-919a-77d22fefd3ce-kube-api-access-8t5vn\") pod \"payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z\" (UID: \"09eb8585-91d8-41a7-919a-77d22fefd3ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z"
Nov 03 19:57:34.917327 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:34.907544    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd388f7-86d4-4f57-8438-342a69aff43b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b\" (UID: \"dbd388f7-86d4-4f57-8438-342a69aff43b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b"
Nov 03 19:57:34.967684 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:34.959763    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh7vv\" (UniqueName: \"kubernetes.io/projected/dbd388f7-86d4-4f57-8438-342a69aff43b-kube-api-access-xh7vv\") pod \"payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b\" (UID: \"dbd388f7-86d4-4f57-8438-342a69aff43b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b"
Nov 03 19:57:34.984257 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:34.982135    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b"
Nov 03 19:57:35.051243 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:35.048664    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09eb8585-91d8-41a7-919a-77d22fefd3ce-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z\" (UID: \"09eb8585-91d8-41a7-919a-77d22fefd3ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z"
Nov 03 19:57:35.051243 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:35.048734    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t5vn\" (UniqueName: \"kubernetes.io/projected/09eb8585-91d8-41a7-919a-77d22fefd3ce-kube-api-access-8t5vn\") pod \"payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z\" (UID: \"09eb8585-91d8-41a7-919a-77d22fefd3ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z"
Nov 03 19:57:35.051243 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:35.050357    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09eb8585-91d8-41a7-919a-77d22fefd3ce-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z\" (UID: \"09eb8585-91d8-41a7-919a-77d22fefd3ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z"
Nov 03 19:57:35.121315 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:35.121257    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t5vn\" (UniqueName: \"kubernetes.io/projected/09eb8585-91d8-41a7-919a-77d22fefd3ce-kube-api-access-8t5vn\") pod \"payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z\" (UID: \"09eb8585-91d8-41a7-919a-77d22fefd3ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z"
Nov 03 19:57:35.333497 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:35.333453    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z"
Nov 03 19:57:35.535173 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:57:35.498831    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd388f7_86d4_4f57_8438_342a69aff43b.slice/crio-75f398ccee529c86296bffde46b83f7a8bd554191ce6fcfe8de68e1fe3ba8c39 WatchSource:0}: Error finding container 75f398ccee529c86296bffde46b83f7a8bd554191ce6fcfe8de68e1fe3ba8c39: Status 404 returned error can't find the container with id 75f398ccee529c86296bffde46b83f7a8bd554191ce6fcfe8de68e1fe3ba8c39
Nov 03 19:57:35.535173 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:35.528557    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b"]
Nov 03 19:57:35.626610 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:35.626575    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z"]
Nov 03 19:57:35.627638 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:57:35.627608    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09eb8585_91d8_41a7_919a_77d22fefd3ce.slice/crio-3cfef46fdf566b79b0651de46c0fb3dd1dcef0c9a2317bbfb9a99fba952a9948 WatchSource:0}: Error finding container 3cfef46fdf566b79b0651de46c0fb3dd1dcef0c9a2317bbfb9a99fba952a9948: Status 404 returned error can't find the container with id 3cfef46fdf566b79b0651de46c0fb3dd1dcef0c9a2317bbfb9a99fba952a9948
Nov 03 19:57:36.201129 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:36.201090    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b" event={"ID":"dbd388f7-86d4-4f57-8438-342a69aff43b","Type":"ContainerStarted","Data":"0c5c78176dfc1d3335edb78150068141894d24acdde38ad511a8d4dfc0cd875a"}
Nov 03 19:57:36.201643 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:36.201161    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b" event={"ID":"dbd388f7-86d4-4f57-8438-342a69aff43b","Type":"ContainerStarted","Data":"75f398ccee529c86296bffde46b83f7a8bd554191ce6fcfe8de68e1fe3ba8c39"}
Nov 03 19:57:36.201643 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:36.201335    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b"
Nov 03 19:57:36.202882 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:36.202856    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z" event={"ID":"09eb8585-91d8-41a7-919a-77d22fefd3ce","Type":"ContainerStarted","Data":"02bd4d819ad72cb6dbebebf998c9a888539551fa705d11c24249806c7e287ca0"}
Nov 03 19:57:36.203001 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:36.202895    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z" event={"ID":"09eb8585-91d8-41a7-919a-77d22fefd3ce","Type":"ContainerStarted","Data":"3cfef46fdf566b79b0651de46c0fb3dd1dcef0c9a2317bbfb9a99fba952a9948"}
Nov 03 19:57:36.203001 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:36.202931    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z"
Nov 03 19:57:36.234348 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:36.234295    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b" podStartSLOduration=2.234280531 podStartE2EDuration="2.234280531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:36.229948494 +0000 UTC m=+1205.002241512" watchObservedRunningTime="2026-11-03 19:57:36.234280531 +0000 UTC m=+1205.006573548"
Nov 03 19:57:36.267766 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:36.267714    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z" podStartSLOduration=2.267699957 podStartE2EDuration="2.267699957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:36.26729187 +0000 UTC m=+1205.039584888" watchObservedRunningTime="2026-11-03 19:57:36.267699957 +0000 UTC m=+1205.039993010"
Nov 03 19:57:39.917662 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:39.917618    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"]
Nov 03 19:57:39.922295 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:39.922276    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:39.950154 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:39.950125    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-29f633-openshift-default-dockercfg-9hxr8\""
Nov 03 19:57:39.968141 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:39.968109    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"]
Nov 03 19:57:39.995182 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:39.995146    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2hjs\" (UniqueName: \"kubernetes.io/projected/41a51c01-3bd6-4414-9f7e-6347d2d59727-kube-api-access-z2hjs\") pod \"e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") " pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:39.995358 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:39.995311    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-workload-socket\") pod \"e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") " pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:39.995358 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:39.995349    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-credential-socket\") pod \"e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") " pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:39.995425 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:39.995379    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-token\") pod \"e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") " pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:39.995497 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:39.995480    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-podinfo\") pod \"e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") " pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:39.995536 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:39.995511    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-data\") pod \"e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") " pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:39.995624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:39.995611    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-envoy\") pod \"e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") " pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:39.995755 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:39.995740    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a51c01-3bd6-4414-9f7e-6347d2d59727-istiod-ca-cert\") pod \"e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") " pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:39.995787 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:39.995772    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-workload-certs\") pod \"e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") " pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:40.097317 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:40.097280    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-envoy\") pod \"e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") " pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:40.097489 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:40.097431    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a51c01-3bd6-4414-9f7e-6347d2d59727-istiod-ca-cert\") pod \"e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") " pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:40.097489 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:40.097455    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-workload-certs\") pod \"e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") " pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:40.097489 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:40.097481    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2hjs\" (UniqueName: \"kubernetes.io/projected/41a51c01-3bd6-4414-9f7e-6347d2d59727-kube-api-access-z2hjs\") pod \"e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") " pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:40.097684 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:40.097562    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-workload-socket\") pod \"e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") " pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:40.097684 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:40.097591    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-credential-socket\") pod \"e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") " pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:40.097684 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:40.097631    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-token\") pod \"e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") " pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:40.097848 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:40.097706    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-podinfo\") pod \"e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") " pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:40.097848 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:40.097741    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-data\") pod \"e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") " pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:40.097956 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:40.097924    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-workload-certs\") pod \"e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") " pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:40.098013 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:40.097979    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-workload-socket\") pod \"e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") " pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:40.098111 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:40.098095    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-data\") pod \"e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") " pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:40.098270 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:40.098249    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a51c01-3bd6-4414-9f7e-6347d2d59727-istiod-ca-cert\") pod \"e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") " pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:40.098471 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:40.098452    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-credential-socket\") pod \"e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") " pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:40.099670 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:40.099653    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-envoy\") pod \"e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") " pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:40.100553 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:40.100526    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-podinfo\") pod \"e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") " pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:40.136978 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:40.136947    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-token\") pod \"e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") " pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:40.141633 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:40.141603    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2hjs\" (UniqueName: \"kubernetes.io/projected/41a51c01-3bd6-4414-9f7e-6347d2d59727-kube-api-access-z2hjs\") pod \"e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") " pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:40.244279 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:40.243721    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:40.703612 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:40.703585    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"]
Nov 03 19:57:40.704816 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:57:40.704790    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a51c01_3bd6_4414_9f7e_6347d2d59727.slice/crio-7c060aac16400f4d3a8836f3de3163355cbedafa9161e6f4657b3a202042ba47 WatchSource:0}: Error finding container 7c060aac16400f4d3a8836f3de3163355cbedafa9161e6f4657b3a202042ba47: Status 404 returned error can't find the container with id 7c060aac16400f4d3a8836f3de3163355cbedafa9161e6f4657b3a202042ba47
Nov 03 19:57:40.707545 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:40.707511    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:57:40.707628 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:40.707577    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:57:40.707628 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:40.707605    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:57:41.325068 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:41.325028    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx" event={"ID":"41a51c01-3bd6-4414-9f7e-6347d2d59727","Type":"ContainerStarted","Data":"207a93fb3a8b4488f693bbc0b80e459f5e849c83c45704d7787db623441feff0"}
Nov 03 19:57:41.325487 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:41.325086    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx" event={"ID":"41a51c01-3bd6-4414-9f7e-6347d2d59727","Type":"ContainerStarted","Data":"7c060aac16400f4d3a8836f3de3163355cbedafa9161e6f4657b3a202042ba47"}
Nov 03 19:57:41.359558 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:41.359439    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx" podStartSLOduration=2.359420874 podStartE2EDuration="2.359420874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:41.357055766 +0000 UTC m=+1210.129348785" watchObservedRunningTime="2026-11-03 19:57:41.359420874 +0000 UTC m=+1210.131713893"
Nov 03 19:57:41.940806 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:41.931258    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln"]
Nov 03 19:57:41.940806 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:41.931705    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln" podUID="44e4835d-2ca1-4b1f-bcec-4095b41b6420" containerName="maas-api" containerID="cri-o://8ebc1e8052f16fb159e317b34b42ead1b212f594eb06f1447ad887c3b7c278cd" gracePeriod=30
Nov 03 19:57:42.291289 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:42.228838    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5"
Nov 03 19:57:42.291289 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:42.257713    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:42.312227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:42.292044    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7"
Nov 03 19:57:42.406236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:42.388870    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:42.510143 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:42.509865    5591 generic.go:356] "Generic (PLEG): container finished" podID="44e4835d-2ca1-4b1f-bcec-4095b41b6420" containerID="8ebc1e8052f16fb159e317b34b42ead1b212f594eb06f1447ad887c3b7c278cd" exitCode=0
Nov 03 19:57:42.510143 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:42.510129    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln" event={"ID":"44e4835d-2ca1-4b1f-bcec-4095b41b6420","Type":"ContainerDied","Data":"8ebc1e8052f16fb159e317b34b42ead1b212f594eb06f1447ad887c3b7c278cd"}
Nov 03 19:57:42.510698 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:42.510679    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:42.512090 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:42.512068    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:57:42.590974 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:42.590945    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln"
Nov 03 19:57:42.703480 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:42.703449    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44e4835d-2ca1-4b1f-bcec-4095b41b6420-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44e4835d-2ca1-4b1f-bcec-4095b41b6420-openshift-service-ca\") pod \"44e4835d-2ca1-4b1f-bcec-4095b41b6420\" (UID: \"44e4835d-2ca1-4b1f-bcec-4095b41b6420\") "
Nov 03 19:57:42.703655 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:42.703513    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44e4835d-2ca1-4b1f-bcec-4095b41b6420-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44e4835d-2ca1-4b1f-bcec-4095b41b6420-maas-api-tls\") pod \"44e4835d-2ca1-4b1f-bcec-4095b41b6420\" (UID: \"44e4835d-2ca1-4b1f-bcec-4095b41b6420\") "
Nov 03 19:57:42.703696 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:42.703653    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44e4835d-2ca1-4b1f-bcec-4095b41b6420-metrics-certs\" (UniqueName: \"kubernetes.io/secret/44e4835d-2ca1-4b1f-bcec-4095b41b6420-metrics-certs\") pod \"44e4835d-2ca1-4b1f-bcec-4095b41b6420\" (UID: \"44e4835d-2ca1-4b1f-bcec-4095b41b6420\") "
Nov 03 19:57:42.703847 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:42.703830    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44e4835d-2ca1-4b1f-bcec-4095b41b6420-kube-api-access-79nhf\" (UniqueName: \"kubernetes.io/projected/44e4835d-2ca1-4b1f-bcec-4095b41b6420-kube-api-access-79nhf\") pod \"44e4835d-2ca1-4b1f-bcec-4095b41b6420\" (UID: \"44e4835d-2ca1-4b1f-bcec-4095b41b6420\") "
Nov 03 19:57:42.703911 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:42.703858    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44e4835d-2ca1-4b1f-bcec-4095b41b6420-openshift-service-ca" pod "44e4835d-2ca1-4b1f-bcec-4095b41b6420" (UID: "44e4835d-2ca1-4b1f-bcec-4095b41b6420"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:57:42.704708 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:42.704684    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44e4835d-2ca1-4b1f-bcec-4095b41b6420-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:42.705839 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:42.705815    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44e4835d-2ca1-4b1f-bcec-4095b41b6420-maas-api-tls" pod "44e4835d-2ca1-4b1f-bcec-4095b41b6420" (UID: "44e4835d-2ca1-4b1f-bcec-4095b41b6420"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:57:42.705919 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:42.705870    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44e4835d-2ca1-4b1f-bcec-4095b41b6420-metrics-certs" pod "44e4835d-2ca1-4b1f-bcec-4095b41b6420" (UID: "44e4835d-2ca1-4b1f-bcec-4095b41b6420"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:57:42.706043 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:42.706028    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44e4835d-2ca1-4b1f-bcec-4095b41b6420-kube-api-access-79nhf" pod "44e4835d-2ca1-4b1f-bcec-4095b41b6420" (UID: "44e4835d-2ca1-4b1f-bcec-4095b41b6420"). InnerVolumeSpecName "kube-api-access-79nhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:57:42.813280 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:42.806675    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79nhf\" (UniqueName: \"kubernetes.io/projected/44e4835d-2ca1-4b1f-bcec-4095b41b6420-kube-api-access-79nhf\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:42.813280 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:42.806728    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44e4835d-2ca1-4b1f-bcec-4095b41b6420-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:42.813280 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:42.806743    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44e4835d-2ca1-4b1f-bcec-4095b41b6420-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:57:43.518976 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:43.518941    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln"
Nov 03 19:57:43.518976 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:43.518957    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln" event={"ID":"44e4835d-2ca1-4b1f-bcec-4095b41b6420","Type":"ContainerDied","Data":"547f6927ce23f7a4491f22529b7f88f05055a7a4e07fd7e66b02c5c44809a972"}
Nov 03 19:57:43.519621 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:43.519026    5591 scope.go:122] "RemoveContainer" containerID="8ebc1e8052f16fb159e317b34b42ead1b212f594eb06f1447ad887c3b7c278cd"
Nov 03 19:57:43.539924 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:43.539897    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln"]
Nov 03 19:57:43.543679 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:43.543655    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9a6d4b-845bc7c595-28pln"]
Nov 03 19:57:45.359896 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:45.359853    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44e4835d-2ca1-4b1f-bcec-4095b41b6420" path="/var/lib/kubelet/pods/44e4835d-2ca1-4b1f-bcec-4095b41b6420/volumes"
Nov 03 19:57:46.164359 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:46.164326    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g"]
Nov 03 19:57:46.165554 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:46.165528    5591 cpu_manager.go:414] "containerMap: removing container" podUID="44e4835d-2ca1-4b1f-bcec-4095b41b6420" containerName="maas-api"
Nov 03 19:57:46.165554 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:46.165554    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44e4835d-2ca1-4b1f-bcec-4095b41b6420" containerName="maas-api"
Nov 03 19:57:46.165786 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:46.165771    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="44e4835d-2ca1-4b1f-bcec-4095b41b6420" containerName="maas-api"
Nov 03 19:57:46.172107 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:46.172082    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g"
Nov 03 19:57:46.185704 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:46.185672    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3b6e2b-serving-cert\""
Nov 03 19:57:46.190077 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:46.190046    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g"]
Nov 03 19:57:46.300435 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:46.300397    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6fcj\" (UniqueName: \"kubernetes.io/projected/79b1ecb5-d452-4402-a4bf-9e89e9e67131-kube-api-access-p6fcj\") pod \"maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g\" (UID: \"79b1ecb5-d452-4402-a4bf-9e89e9e67131\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g"
Nov 03 19:57:46.300631 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:46.300541    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b1ecb5-d452-4402-a4bf-9e89e9e67131-metrics-certs\") pod \"maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g\" (UID: \"79b1ecb5-d452-4402-a4bf-9e89e9e67131\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g"
Nov 03 19:57:46.300631 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:46.300619    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b1ecb5-d452-4402-a4bf-9e89e9e67131-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g\" (UID: \"79b1ecb5-d452-4402-a4bf-9e89e9e67131\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g"
Nov 03 19:57:46.300832 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:46.300812    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b1ecb5-d452-4402-a4bf-9e89e9e67131-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g\" (UID: \"79b1ecb5-d452-4402-a4bf-9e89e9e67131\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g"
Nov 03 19:57:46.415289 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:46.414263    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b1ecb5-d452-4402-a4bf-9e89e9e67131-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g\" (UID: \"79b1ecb5-d452-4402-a4bf-9e89e9e67131\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g"
Nov 03 19:57:46.415289 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:46.414541    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b1ecb5-d452-4402-a4bf-9e89e9e67131-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g\" (UID: \"79b1ecb5-d452-4402-a4bf-9e89e9e67131\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g"
Nov 03 19:57:46.415289 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:46.414717    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6fcj\" (UniqueName: \"kubernetes.io/projected/79b1ecb5-d452-4402-a4bf-9e89e9e67131-kube-api-access-p6fcj\") pod \"maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g\" (UID: \"79b1ecb5-d452-4402-a4bf-9e89e9e67131\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g"
Nov 03 19:57:46.415289 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:46.414811    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b1ecb5-d452-4402-a4bf-9e89e9e67131-metrics-certs\") pod \"maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g\" (UID: \"79b1ecb5-d452-4402-a4bf-9e89e9e67131\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g"
Nov 03 19:57:46.448231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:46.430632    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b1ecb5-d452-4402-a4bf-9e89e9e67131-metrics-certs\") pod \"maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g\" (UID: \"79b1ecb5-d452-4402-a4bf-9e89e9e67131\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g"
Nov 03 19:57:46.448965 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:46.431136    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b1ecb5-d452-4402-a4bf-9e89e9e67131-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g\" (UID: \"79b1ecb5-d452-4402-a4bf-9e89e9e67131\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g"
Nov 03 19:57:46.451941 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:46.451913    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b1ecb5-d452-4402-a4bf-9e89e9e67131-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g\" (UID: \"79b1ecb5-d452-4402-a4bf-9e89e9e67131\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g"
Nov 03 19:57:46.452117 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:46.452027    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6fcj\" (UniqueName: \"kubernetes.io/projected/79b1ecb5-d452-4402-a4bf-9e89e9e67131-kube-api-access-p6fcj\") pod \"maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g\" (UID: \"79b1ecb5-d452-4402-a4bf-9e89e9e67131\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g"
Nov 03 19:57:46.500231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:46.499501    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g"
Nov 03 19:57:46.893188 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:46.893124    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g"]
Nov 03 19:57:47.216957 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:47.216878    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z"
Nov 03 19:57:47.216957 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:47.216947    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b"
Nov 03 19:57:47.873787 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:47.873748    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g" event={"ID":"79b1ecb5-d452-4402-a4bf-9e89e9e67131","Type":"ContainerStarted","Data":"8951af66b1f169247e4c762c57f0f3da58f489ec07b4f7197d2ed250240edee9"}
Nov 03 19:57:47.874288 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:47.873801    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g" event={"ID":"79b1ecb5-d452-4402-a4bf-9e89e9e67131","Type":"ContainerStarted","Data":"9765c2e8b7506824f681f3babb1a506c7b470a9beb76bde20fcee413b83ae83a"}
Nov 03 19:57:47.874288 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:47.873833    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g"
Nov 03 19:57:47.898999 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:47.898941    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g" podStartSLOduration=1.118625212 podStartE2EDuration="1.898925935s" totalImagesPullingTime="780.300723ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:47.894469213 +0000 UTC m=+1216.666762232" watchObservedRunningTime="2026-11-03 19:57:47.898925935 +0000 UTC m=+1216.671218952"
Nov 03 19:57:53.896237 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:53.896166    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g"
Nov 03 19:57:58.066798 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:58.066759    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2"]
Nov 03 19:57:58.073903 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:58.073876    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2"
Nov 03 19:57:58.084957 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:58.084925    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2"]
Nov 03 19:57:58.160306 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:58.160276    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-1b1d04-dockercfg-kjs5n\""
Nov 03 19:57:58.170712 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:58.170676    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vggnt\" (UniqueName: \"kubernetes.io/projected/836a78aa-c8d3-47c2-8b42-3bfd919fa477-kube-api-access-vggnt\") pod \"payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2\" (UID: \"836a78aa-c8d3-47c2-8b42-3bfd919fa477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2"
Nov 03 19:57:58.170876 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:58.170795    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/836a78aa-c8d3-47c2-8b42-3bfd919fa477-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2\" (UID: \"836a78aa-c8d3-47c2-8b42-3bfd919fa477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2"
Nov 03 19:57:58.171544 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:58.171521    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-1b1d04\""
Nov 03 19:57:58.288230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:58.279608    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vggnt\" (UniqueName: \"kubernetes.io/projected/836a78aa-c8d3-47c2-8b42-3bfd919fa477-kube-api-access-vggnt\") pod \"payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2\" (UID: \"836a78aa-c8d3-47c2-8b42-3bfd919fa477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2"
Nov 03 19:57:58.288230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:58.279748    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/836a78aa-c8d3-47c2-8b42-3bfd919fa477-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2\" (UID: \"836a78aa-c8d3-47c2-8b42-3bfd919fa477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2"
Nov 03 19:57:58.299653 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:58.280929    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/836a78aa-c8d3-47c2-8b42-3bfd919fa477-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2\" (UID: \"836a78aa-c8d3-47c2-8b42-3bfd919fa477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2"
Nov 03 19:57:58.430289 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:58.421298    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vggnt\" (UniqueName: \"kubernetes.io/projected/836a78aa-c8d3-47c2-8b42-3bfd919fa477-kube-api-access-vggnt\") pod \"payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2\" (UID: \"836a78aa-c8d3-47c2-8b42-3bfd919fa477\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2"
Nov 03 19:57:58.694792 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:58.694687    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2"
Nov 03 19:57:58.937230 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:57:58.933375    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:57:59.320853 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:57:59.320814    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2"]
Nov 03 19:58:00.020671 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:00.020633    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2" event={"ID":"836a78aa-c8d3-47c2-8b42-3bfd919fa477","Type":"ContainerStarted","Data":"5ec71ebb619cbe952e9d9ca2c58e395ad74d10b9a157b2a1480f77f7167cad86"}
Nov 03 19:58:00.020856 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:00.020697    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2" event={"ID":"836a78aa-c8d3-47c2-8b42-3bfd919fa477","Type":"ContainerStarted","Data":"5290978dceb5a2a42635aecb056c5b639e0a2f29f8fc7b395054f216caf1d9cb"}
Nov 03 19:58:00.020856 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:00.020788    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2"
Nov 03 19:58:00.046357 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:00.046298    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2" podStartSLOduration=2.04628081 podStartE2EDuration="2.04628081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:00.043387263 +0000 UTC m=+1228.815680281" watchObservedRunningTime="2026-11-03 19:58:00.04628081 +0000 UTC m=+1228.818573832"
Nov 03 19:58:02.779442 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:02.779404    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z"]
Nov 03 19:58:02.793499 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:02.793461    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b"]
Nov 03 19:58:07.260781 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:07.260743    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:58:07.780301 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:07.780240    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z" podUID="09eb8585-91d8-41a7-919a-77d22fefd3ce" containerName="payload-processing" containerID="cri-o://02bd4d819ad72cb6dbebebf998c9a888539551fa705d11c24249806c7e287ca0" gracePeriod=25
Nov 03 19:58:07.793881 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:07.793849    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b" podUID="dbd388f7-86d4-4f57-8438-342a69aff43b" containerName="payload-pre-processing" containerID="cri-o://0c5c78176dfc1d3335edb78150068141894d24acdde38ad511a8d4dfc0cd875a" gracePeriod=25
Nov 03 19:58:08.111321 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:08.111284    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z"
Nov 03 19:58:08.118281 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:08.118250    5591 generic.go:356] "Generic (PLEG): container finished" podID="dbd388f7-86d4-4f57-8438-342a69aff43b" containerID="0c5c78176dfc1d3335edb78150068141894d24acdde38ad511a8d4dfc0cd875a" exitCode=0
Nov 03 19:58:08.118435 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:08.118320    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b" event={"ID":"dbd388f7-86d4-4f57-8438-342a69aff43b","Type":"ContainerDied","Data":"0c5c78176dfc1d3335edb78150068141894d24acdde38ad511a8d4dfc0cd875a"}
Nov 03 19:58:08.120154 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:08.120128    5591 generic.go:356] "Generic (PLEG): container finished" podID="09eb8585-91d8-41a7-919a-77d22fefd3ce" containerID="02bd4d819ad72cb6dbebebf998c9a888539551fa705d11c24249806c7e287ca0" exitCode=0
Nov 03 19:58:08.120344 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:08.120192    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z"
Nov 03 19:58:08.120344 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:08.120235    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z" event={"ID":"09eb8585-91d8-41a7-919a-77d22fefd3ce","Type":"ContainerDied","Data":"02bd4d819ad72cb6dbebebf998c9a888539551fa705d11c24249806c7e287ca0"}
Nov 03 19:58:08.120344 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:08.120293    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z" event={"ID":"09eb8585-91d8-41a7-919a-77d22fefd3ce","Type":"ContainerDied","Data":"3cfef46fdf566b79b0651de46c0fb3dd1dcef0c9a2317bbfb9a99fba952a9948"}
Nov 03 19:58:08.120524 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:08.120325    5591 scope.go:122] "RemoveContainer" containerID="02bd4d819ad72cb6dbebebf998c9a888539551fa705d11c24249806c7e287ca0"
Nov 03 19:58:08.266230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:08.223401    5591 scope.go:122] "RemoveContainer" containerID="02bd4d819ad72cb6dbebebf998c9a888539551fa705d11c24249806c7e287ca0"
Nov 03 19:58:08.266230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:08.254563    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09eb8585-91d8-41a7-919a-77d22fefd3ce-kube-api-access-8t5vn\" (UniqueName: \"kubernetes.io/projected/09eb8585-91d8-41a7-919a-77d22fefd3ce-kube-api-access-8t5vn\") pod \"09eb8585-91d8-41a7-919a-77d22fefd3ce\" (UID: \"09eb8585-91d8-41a7-919a-77d22fefd3ce\") "
Nov 03 19:58:08.266230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:08.254695    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09eb8585-91d8-41a7-919a-77d22fefd3ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09eb8585-91d8-41a7-919a-77d22fefd3ce-plugins-config-volume\") pod \"09eb8585-91d8-41a7-919a-77d22fefd3ce\" (UID: \"09eb8585-91d8-41a7-919a-77d22fefd3ce\") "
Nov 03 19:58:08.333229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:08.327944    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09eb8585-91d8-41a7-919a-77d22fefd3ce-plugins-config-volume" pod "09eb8585-91d8-41a7-919a-77d22fefd3ce" (UID: "09eb8585-91d8-41a7-919a-77d22fefd3ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:58:08.350733 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:08.349462    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09eb8585-91d8-41a7-919a-77d22fefd3ce-kube-api-access-8t5vn" pod "09eb8585-91d8-41a7-919a-77d22fefd3ce" (UID: "09eb8585-91d8-41a7-919a-77d22fefd3ce"). InnerVolumeSpecName "kube-api-access-8t5vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:58:08.350733 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:58:08.350577    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02bd4d819ad72cb6dbebebf998c9a888539551fa705d11c24249806c7e287ca0\": container with ID starting with 02bd4d819ad72cb6dbebebf998c9a888539551fa705d11c24249806c7e287ca0 not found: ID does not exist" containerID="02bd4d819ad72cb6dbebebf998c9a888539551fa705d11c24249806c7e287ca0"
Nov 03 19:58:08.350733 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:08.350617    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02bd4d819ad72cb6dbebebf998c9a888539551fa705d11c24249806c7e287ca0"} err="failed to get container status \"02bd4d819ad72cb6dbebebf998c9a888539551fa705d11c24249806c7e287ca0\": rpc error: code = NotFound desc = could not find container \"02bd4d819ad72cb6dbebebf998c9a888539551fa705d11c24249806c7e287ca0\": container with ID starting with 02bd4d819ad72cb6dbebebf998c9a888539551fa705d11c24249806c7e287ca0 not found: ID does not exist"
Nov 03 19:58:08.441120 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:08.441055    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t5vn\" (UniqueName: \"kubernetes.io/projected/09eb8585-91d8-41a7-919a-77d22fefd3ce-kube-api-access-8t5vn\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:08.441841 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:08.441729    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09eb8585-91d8-41a7-919a-77d22fefd3ce-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:08.454143 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:08.453815    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b"
Nov 03 19:58:08.512445 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:08.512393    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z"]
Nov 03 19:58:08.519133 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:08.519095    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3b6e2b-b9f688cd8-24w9z"]
Nov 03 19:58:08.554921 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:08.554864    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd388f7-86d4-4f57-8438-342a69aff43b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd388f7-86d4-4f57-8438-342a69aff43b-plugins-config-volume\") pod \"dbd388f7-86d4-4f57-8438-342a69aff43b\" (UID: \"dbd388f7-86d4-4f57-8438-342a69aff43b\") "
Nov 03 19:58:08.555540 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:08.555512    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd388f7-86d4-4f57-8438-342a69aff43b-kube-api-access-xh7vv\" (UniqueName: \"kubernetes.io/projected/dbd388f7-86d4-4f57-8438-342a69aff43b-kube-api-access-xh7vv\") pod \"dbd388f7-86d4-4f57-8438-342a69aff43b\" (UID: \"dbd388f7-86d4-4f57-8438-342a69aff43b\") "
Nov 03 19:58:08.558738 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:08.558706    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd388f7-86d4-4f57-8438-342a69aff43b-plugins-config-volume" pod "dbd388f7-86d4-4f57-8438-342a69aff43b" (UID: "dbd388f7-86d4-4f57-8438-342a69aff43b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:58:08.567257 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:08.566353    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd388f7-86d4-4f57-8438-342a69aff43b-kube-api-access-xh7vv" pod "dbd388f7-86d4-4f57-8438-342a69aff43b" (UID: "dbd388f7-86d4-4f57-8438-342a69aff43b"). InnerVolumeSpecName "kube-api-access-xh7vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:58:08.658790 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:08.658706    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh7vv\" (UniqueName: \"kubernetes.io/projected/dbd388f7-86d4-4f57-8438-342a69aff43b-kube-api-access-xh7vv\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:08.658790 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:08.658735    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd388f7-86d4-4f57-8438-342a69aff43b-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:09.137828 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.137798    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b"
Nov 03 19:58:09.138568 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.138540    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b" event={"ID":"dbd388f7-86d4-4f57-8438-342a69aff43b","Type":"ContainerDied","Data":"75f398ccee529c86296bffde46b83f7a8bd554191ce6fcfe8de68e1fe3ba8c39"}
Nov 03 19:58:09.138918 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.138602    5591 scope.go:122] "RemoveContainer" containerID="0c5c78176dfc1d3335edb78150068141894d24acdde38ad511a8d4dfc0cd875a"
Nov 03 19:58:09.163179 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.163151    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b"]
Nov 03 19:58:09.172329 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.172294    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3b6e2b-bdb96dccf-2zw4b"]
Nov 03 19:58:09.212228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.209129    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w"]
Nov 03 19:58:09.212228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.210250    5591 cpu_manager.go:414] "containerMap: removing container" podUID="09eb8585-91d8-41a7-919a-77d22fefd3ce" containerName="payload-processing"
Nov 03 19:58:09.212228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.210270    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09eb8585-91d8-41a7-919a-77d22fefd3ce" containerName="payload-processing"
Nov 03 19:58:09.212228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.210294    5591 cpu_manager.go:414] "containerMap: removing container" podUID="dbd388f7-86d4-4f57-8438-342a69aff43b" containerName="payload-pre-processing"
Nov 03 19:58:09.212228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.210301    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd388f7-86d4-4f57-8438-342a69aff43b" containerName="payload-pre-processing"
Nov 03 19:58:09.212228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.210518    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="09eb8585-91d8-41a7-919a-77d22fefd3ce" containerName="payload-processing"
Nov 03 19:58:09.212228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.210537    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd388f7-86d4-4f57-8438-342a69aff43b" containerName="payload-pre-processing"
Nov 03 19:58:09.243468 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.243433    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w"]
Nov 03 19:58:09.243802 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.243787    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w"
Nov 03 19:58:09.251094 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.250442    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-1b1d04-serving-cert\""
Nov 03 19:58:09.268166 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.268051    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6377f5-2c6f-4005-bac4-9caba84e3314-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w\" (UID: \"7a6377f5-2c6f-4005-bac4-9caba84e3314\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w"
Nov 03 19:58:09.268166 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.268129    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a6377f5-2c6f-4005-bac4-9caba84e3314-metrics-certs\") pod \"maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w\" (UID: \"7a6377f5-2c6f-4005-bac4-9caba84e3314\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w"
Nov 03 19:58:09.268723 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.268228    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a6377f5-2c6f-4005-bac4-9caba84e3314-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w\" (UID: \"7a6377f5-2c6f-4005-bac4-9caba84e3314\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w"
Nov 03 19:58:09.268723 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.268609    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtgmp\" (UniqueName: \"kubernetes.io/projected/7a6377f5-2c6f-4005-bac4-9caba84e3314-kube-api-access-mtgmp\") pod \"maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w\" (UID: \"7a6377f5-2c6f-4005-bac4-9caba84e3314\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w"
Nov 03 19:58:09.354249 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.354185    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09eb8585-91d8-41a7-919a-77d22fefd3ce" path="/var/lib/kubelet/pods/09eb8585-91d8-41a7-919a-77d22fefd3ce/volumes"
Nov 03 19:58:09.354752 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.354717    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbd388f7-86d4-4f57-8438-342a69aff43b" path="/var/lib/kubelet/pods/dbd388f7-86d4-4f57-8438-342a69aff43b/volumes"
Nov 03 19:58:09.370418 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.370386    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6377f5-2c6f-4005-bac4-9caba84e3314-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w\" (UID: \"7a6377f5-2c6f-4005-bac4-9caba84e3314\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w"
Nov 03 19:58:09.370593 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.370430    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a6377f5-2c6f-4005-bac4-9caba84e3314-metrics-certs\") pod \"maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w\" (UID: \"7a6377f5-2c6f-4005-bac4-9caba84e3314\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w"
Nov 03 19:58:09.370593 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.370467    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a6377f5-2c6f-4005-bac4-9caba84e3314-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w\" (UID: \"7a6377f5-2c6f-4005-bac4-9caba84e3314\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w"
Nov 03 19:58:09.370721 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.370695    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtgmp\" (UniqueName: \"kubernetes.io/projected/7a6377f5-2c6f-4005-bac4-9caba84e3314-kube-api-access-mtgmp\") pod \"maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w\" (UID: \"7a6377f5-2c6f-4005-bac4-9caba84e3314\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w"
Nov 03 19:58:09.371131 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.371105    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6377f5-2c6f-4005-bac4-9caba84e3314-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w\" (UID: \"7a6377f5-2c6f-4005-bac4-9caba84e3314\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w"
Nov 03 19:58:09.373075 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.373051    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a6377f5-2c6f-4005-bac4-9caba84e3314-metrics-certs\") pod \"maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w\" (UID: \"7a6377f5-2c6f-4005-bac4-9caba84e3314\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w"
Nov 03 19:58:09.373159 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.373100    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a6377f5-2c6f-4005-bac4-9caba84e3314-maas-api-tls\") pod \"maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w\" (UID: \"7a6377f5-2c6f-4005-bac4-9caba84e3314\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w"
Nov 03 19:58:09.461318 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.454020    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtgmp\" (UniqueName: \"kubernetes.io/projected/7a6377f5-2c6f-4005-bac4-9caba84e3314-kube-api-access-mtgmp\") pod \"maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w\" (UID: \"7a6377f5-2c6f-4005-bac4-9caba84e3314\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w"
Nov 03 19:58:09.585230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:09.584876    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w"
Nov 03 19:58:10.553095 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:10.541822    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w"]
Nov 03 19:58:11.049241 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:11.048737    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2"
Nov 03 19:58:11.366562 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:11.366526    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w" event={"ID":"7a6377f5-2c6f-4005-bac4-9caba84e3314","Type":"ContainerStarted","Data":"3b48a4d6be3fc893ef9721bf659323956e5f64fc54edf33030603d4c0ab12feb"}
Nov 03 19:58:12.372214 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:12.372165    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w" event={"ID":"7a6377f5-2c6f-4005-bac4-9caba84e3314","Type":"ContainerStarted","Data":"1642f1354002650aacbb3ae1237a1364d0ffe77e7944a0d268638b662e5767db"}
Nov 03 19:58:12.372653 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:12.372452    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w"
Nov 03 19:58:12.436090 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:12.436023    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w" podStartSLOduration=2.628431133 podStartE2EDuration="3.436004645s" totalImagesPullingTime="807.573512ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:12.434442119 +0000 UTC m=+1241.206735140" watchObservedRunningTime="2026-11-03 19:58:12.436004645 +0000 UTC m=+1241.208297664"
Nov 03 19:58:15.627708 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:15.625320    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5"]
Nov 03 19:58:15.627708 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:15.625562    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5" podUID="b6d39933-bb86-4a0a-ab3f-bd7265c500af" containerName="maas-api" containerID="cri-o://c1687ec5f8d0756b7b72e86c7a38ec932515d730d5d6e0970a61e9bf037d090e" gracePeriod=30
Nov 03 19:58:16.241699 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:16.241672    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5"
Nov 03 19:58:16.414227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:16.410342    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6d39933-bb86-4a0a-ab3f-bd7265c500af-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6d39933-bb86-4a0a-ab3f-bd7265c500af-metrics-certs\") pod \"b6d39933-bb86-4a0a-ab3f-bd7265c500af\" (UID: \"b6d39933-bb86-4a0a-ab3f-bd7265c500af\") "
Nov 03 19:58:16.414227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:16.410640    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6d39933-bb86-4a0a-ab3f-bd7265c500af-kube-api-access-27q4c\" (UniqueName: \"kubernetes.io/projected/b6d39933-bb86-4a0a-ab3f-bd7265c500af-kube-api-access-27q4c\") pod \"b6d39933-bb86-4a0a-ab3f-bd7265c500af\" (UID: \"b6d39933-bb86-4a0a-ab3f-bd7265c500af\") "
Nov 03 19:58:16.414227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:16.410722    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6d39933-bb86-4a0a-ab3f-bd7265c500af-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6d39933-bb86-4a0a-ab3f-bd7265c500af-maas-api-tls\") pod \"b6d39933-bb86-4a0a-ab3f-bd7265c500af\" (UID: \"b6d39933-bb86-4a0a-ab3f-bd7265c500af\") "
Nov 03 19:58:16.414227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:16.410845    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6d39933-bb86-4a0a-ab3f-bd7265c500af-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d39933-bb86-4a0a-ab3f-bd7265c500af-openshift-service-ca\") pod \"b6d39933-bb86-4a0a-ab3f-bd7265c500af\" (UID: \"b6d39933-bb86-4a0a-ab3f-bd7265c500af\") "
Nov 03 19:58:16.435311 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:16.426491    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d39933-bb86-4a0a-ab3f-bd7265c500af-openshift-service-ca" pod "b6d39933-bb86-4a0a-ab3f-bd7265c500af" (UID: "b6d39933-bb86-4a0a-ab3f-bd7265c500af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:58:16.481982 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:16.481929    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d39933-bb86-4a0a-ab3f-bd7265c500af-kube-api-access-27q4c" pod "b6d39933-bb86-4a0a-ab3f-bd7265c500af" (UID: "b6d39933-bb86-4a0a-ab3f-bd7265c500af"). InnerVolumeSpecName "kube-api-access-27q4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:58:16.482768 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:16.482669    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6d39933-bb86-4a0a-ab3f-bd7265c500af-maas-api-tls" pod "b6d39933-bb86-4a0a-ab3f-bd7265c500af" (UID: "b6d39933-bb86-4a0a-ab3f-bd7265c500af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:58:16.493512 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:16.493461    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6d39933-bb86-4a0a-ab3f-bd7265c500af-metrics-certs" pod "b6d39933-bb86-4a0a-ab3f-bd7265c500af" (UID: "b6d39933-bb86-4a0a-ab3f-bd7265c500af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:58:16.507259 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:16.507220    5591 generic.go:356] "Generic (PLEG): container finished" podID="b6d39933-bb86-4a0a-ab3f-bd7265c500af" containerID="c1687ec5f8d0756b7b72e86c7a38ec932515d730d5d6e0970a61e9bf037d090e" exitCode=0
Nov 03 19:58:16.507415 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:16.507266    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5"
Nov 03 19:58:16.507415 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:16.507300    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5" event={"ID":"b6d39933-bb86-4a0a-ab3f-bd7265c500af","Type":"ContainerDied","Data":"c1687ec5f8d0756b7b72e86c7a38ec932515d730d5d6e0970a61e9bf037d090e"}
Nov 03 19:58:16.507415 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:16.507369    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5" event={"ID":"b6d39933-bb86-4a0a-ab3f-bd7265c500af","Type":"ContainerDied","Data":"f181c872ccb2b9e7a0fea30f20f3009b8a30e9d0d4bda5dbfed5e3afb794a5f0"}
Nov 03 19:58:16.507415 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:16.507402    5591 scope.go:122] "RemoveContainer" containerID="c1687ec5f8d0756b7b72e86c7a38ec932515d730d5d6e0970a61e9bf037d090e"
Nov 03 19:58:16.513347 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:16.513197    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27q4c\" (UniqueName: \"kubernetes.io/projected/b6d39933-bb86-4a0a-ab3f-bd7265c500af-kube-api-access-27q4c\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:16.513347 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:16.513329    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6d39933-bb86-4a0a-ab3f-bd7265c500af-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:16.513347 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:16.513346    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6d39933-bb86-4a0a-ab3f-bd7265c500af-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:16.513585 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:16.513361    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6d39933-bb86-4a0a-ab3f-bd7265c500af-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:16.523730 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:16.523703    5591 scope.go:122] "RemoveContainer" containerID="c1687ec5f8d0756b7b72e86c7a38ec932515d730d5d6e0970a61e9bf037d090e"
Nov 03 19:58:16.524149 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:58:16.524125    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1687ec5f8d0756b7b72e86c7a38ec932515d730d5d6e0970a61e9bf037d090e\": container with ID starting with c1687ec5f8d0756b7b72e86c7a38ec932515d730d5d6e0970a61e9bf037d090e not found: ID does not exist" containerID="c1687ec5f8d0756b7b72e86c7a38ec932515d730d5d6e0970a61e9bf037d090e"
Nov 03 19:58:16.524300 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:16.524157    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1687ec5f8d0756b7b72e86c7a38ec932515d730d5d6e0970a61e9bf037d090e"} err="failed to get container status \"c1687ec5f8d0756b7b72e86c7a38ec932515d730d5d6e0970a61e9bf037d090e\": rpc error: code = NotFound desc = could not find container \"c1687ec5f8d0756b7b72e86c7a38ec932515d730d5d6e0970a61e9bf037d090e\": container with ID starting with c1687ec5f8d0756b7b72e86c7a38ec932515d730d5d6e0970a61e9bf037d090e not found: ID does not exist"
Nov 03 19:58:16.539084 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:16.539047    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5"]
Nov 03 19:58:16.544956 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:16.544928    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6cd7f4-5886cf56ff-4tkj5"]
Nov 03 19:58:17.051405 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:17.051361    5591 patch_prober.go:29] interesting pod/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:58:17.051929 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:17.051436    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq" podUID="f226a9f7-2809-4890-8cc3-29013e49d774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:17.424340 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:17.417246    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6d39933-bb86-4a0a-ab3f-bd7265c500af" path="/var/lib/kubelet/pods/b6d39933-bb86-4a0a-ab3f-bd7265c500af/volumes"
Nov 03 19:58:18.412707 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:18.412674    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w"
Nov 03 19:58:21.687762 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:21.687727    5591 generic.go:356] "Generic (PLEG): container finished" podID="f226a9f7-2809-4890-8cc3-29013e49d774" containerID="8842dc2ff9cb2bc31ba58bbcedbd7f0d914cd991a91fc3b5c2352b06cdb48090" exitCode=0
Nov 03 19:58:21.688254 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:21.687797    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq" event={"ID":"f226a9f7-2809-4890-8cc3-29013e49d774","Type":"ContainerDied","Data":"8842dc2ff9cb2bc31ba58bbcedbd7f0d914cd991a91fc3b5c2352b06cdb48090"}
Nov 03 19:58:21.688403 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:21.688383    5591 scope.go:122] "RemoveContainer" containerID="8842dc2ff9cb2bc31ba58bbcedbd7f0d914cd991a91fc3b5c2352b06cdb48090"
Nov 03 19:58:21.689849 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:21.689828    5591 generic.go:356] "Generic (PLEG): container finished" podID="41a51c01-3bd6-4414-9f7e-6347d2d59727" containerID="207a93fb3a8b4488f693bbc0b80e459f5e849c83c45704d7787db623441feff0" exitCode=0
Nov 03 19:58:21.689981 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:21.689898    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx" event={"ID":"41a51c01-3bd6-4414-9f7e-6347d2d59727","Type":"ContainerDied","Data":"207a93fb3a8b4488f693bbc0b80e459f5e849c83c45704d7787db623441feff0"}
Nov 03 19:58:21.690338 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:21.690308    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:58:21.690398 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:21.690383    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:58:21.690447 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:21.690393    5591 scope.go:122] "RemoveContainer" containerID="207a93fb3a8b4488f693bbc0b80e459f5e849c83c45704d7787db623441feff0"
Nov 03 19:58:21.690447 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:21.690416    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:58:21.692261 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:21.692235    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:58:21.692332 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:21.692318    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:58:21.692377 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:21.692364    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:58:21.958447 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:21.952080    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr"]
Nov 03 19:58:22.725269 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:22.725231    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq" event={"ID":"f226a9f7-2809-4890-8cc3-29013e49d774","Type":"ContainerStarted","Data":"6a575b26f292570d4f7d08bcbac3c47c245a49ce6835d60ed76f5bf50e1297e5"}
Nov 03 19:58:22.744640 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:22.744599    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx" event={"ID":"41a51c01-3bd6-4414-9f7e-6347d2d59727","Type":"ContainerStarted","Data":"4d63baa9ecee0765a1c52a31926b742572329be1668603073303ae7aaf64ecec"}
Nov 03 19:58:22.949645 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:22.949614    5591 patch_prober.go:29] interesting pod/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 19:58:22.949912 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:22.949895    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq" podUID="f226a9f7-2809-4890-8cc3-29013e49d774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 19:58:22.950015 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:22.949852    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:58:23.244777 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:23.244719    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:58:23.246115 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:23.246090    5591 patch_prober.go:29] interesting pod/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 19:58:23.246279 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:23.246146    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx" podUID="41a51c01-3bd6-4414-9f7e-6347d2d59727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 19:58:23.949514 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:23.949478    5591 patch_prober.go:29] interesting pod/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 19:58:23.950056 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:23.949558    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq" podUID="f226a9f7-2809-4890-8cc3-29013e49d774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 19:58:24.244153 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:24.244060    5591 patch_prober.go:29] interesting pod/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 19:58:24.244153 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:24.244130    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx" podUID="41a51c01-3bd6-4414-9f7e-6347d2d59727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 19:58:24.949912 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:24.949873    5591 patch_prober.go:29] interesting pod/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 19:58:24.950397 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:24.949928    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq" podUID="f226a9f7-2809-4890-8cc3-29013e49d774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 19:58:25.244383 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:25.244119    5591 patch_prober.go:29] interesting pod/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 19:58:25.244383 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:25.244186    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx" podUID="41a51c01-3bd6-4414-9f7e-6347d2d59727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 19:58:25.923583 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:25.898718    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"]
Nov 03 19:58:25.923583 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:25.914859    5591 cpu_manager.go:414] "containerMap: removing container" podUID="b6d39933-bb86-4a0a-ab3f-bd7265c500af" containerName="maas-api"
Nov 03 19:58:25.923583 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:25.914888    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6d39933-bb86-4a0a-ab3f-bd7265c500af" containerName="maas-api"
Nov 03 19:58:25.923583 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:25.915053    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6d39933-bb86-4a0a-ab3f-bd7265c500af" containerName="maas-api"
Nov 03 19:58:25.941593 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:25.941555    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:25.957184 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:25.956340    5591 patch_prober.go:29] interesting pod/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 19:58:25.957184 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:25.956414    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq" podUID="f226a9f7-2809-4890-8cc3-29013e49d774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 19:58:25.960964 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:25.958450    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"]
Nov 03 19:58:25.962390 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:25.961558    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-389e6d-openshift-default-dockercfg-hxvqw\""
Nov 03 19:58:26.041006 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.040971    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-credential-socket\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.041239 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.041153    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-data\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.041239 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.041189    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-workload-certs\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.041370 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.041244    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-token\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.041424 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.041399    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-podinfo\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.041461 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.041448    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-envoy\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.041537 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.041523    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k48j\" (UniqueName: \"kubernetes.io/projected/a08831b1-e194-4e39-b23d-6232d8d9851e-kube-api-access-2k48j\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.041646 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.041630    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-workload-socket\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.041683 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.041671    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a08831b1-e194-4e39-b23d-6232d8d9851e-istiod-ca-cert\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.143092 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.143055    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a08831b1-e194-4e39-b23d-6232d8d9851e-istiod-ca-cert\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.143319 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.143108    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-credential-socket\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.143319 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.143214    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-data\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.143319 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.143236    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-workload-certs\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.143319 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.143257    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-token\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.143319 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.143318    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-podinfo\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.143597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.143341    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-envoy\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.143597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.143390    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k48j\" (UniqueName: \"kubernetes.io/projected/a08831b1-e194-4e39-b23d-6232d8d9851e-kube-api-access-2k48j\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.143597 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.143467    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-workload-socket\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.143757 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.143598    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-credential-socket\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.143757 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.143730    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-workload-socket\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.144282 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.144256    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-workload-certs\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.144407 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.144299    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-data\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.144635 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.144614    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a08831b1-e194-4e39-b23d-6232d8d9851e-istiod-ca-cert\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.146360 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.146342    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-envoy\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.146518 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.146497    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-podinfo\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.156598 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.156564    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-token\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.156753 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.156618    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k48j\" (UniqueName: \"kubernetes.io/projected/a08831b1-e194-4e39-b23d-6232d8d9851e-kube-api-access-2k48j\") pod \"e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") " pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.244859 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.244775    5591 patch_prober.go:29] interesting pod/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 19:58:26.244859 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.244831    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx" podUID="41a51c01-3bd6-4414-9f7e-6347d2d59727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 19:58:26.288232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.284522    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:26.949433 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.949351    5591 patch_prober.go:29] interesting pod/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 19:58:26.949433 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.949399    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq" podUID="f226a9f7-2809-4890-8cc3-29013e49d774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 19:58:26.952906 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:26.952873    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr" podUID="8d4d0a3c-0312-4063-b331-3735c26dcc4c" containerName="payload-pre-processing" containerID="cri-o://a6c8fc09e8dd7b17f9387400c7949ad52f2d61be1ad8cad715c1d7316dd424aa" gracePeriod=25
Nov 03 19:58:27.205602 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:27.185187    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"]
Nov 03 19:58:27.215449 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:27.215395    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:58:27.215593 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:27.215493    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:58:27.215593 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:27.215565    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:58:27.247834 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:27.247386    5591 patch_prober.go:29] interesting pod/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 19:58:27.247834 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:27.247459    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx" podUID="41a51c01-3bd6-4414-9f7e-6347d2d59727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 19:58:27.546417 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:27.546387    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:58:27.756579 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:27.756503    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr"
Nov 03 19:58:27.767178 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:27.767144    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d4d0a3c-0312-4063-b331-3735c26dcc4c-kube-api-access-hwk89\" (UniqueName: \"kubernetes.io/projected/8d4d0a3c-0312-4063-b331-3735c26dcc4c-kube-api-access-hwk89\") pod \"8d4d0a3c-0312-4063-b331-3735c26dcc4c\" (UID: \"8d4d0a3c-0312-4063-b331-3735c26dcc4c\") "
Nov 03 19:58:27.767439 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:27.767418    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d4d0a3c-0312-4063-b331-3735c26dcc4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d4d0a3c-0312-4063-b331-3735c26dcc4c-plugins-config-volume\") pod \"8d4d0a3c-0312-4063-b331-3735c26dcc4c\" (UID: \"8d4d0a3c-0312-4063-b331-3735c26dcc4c\") "
Nov 03 19:58:27.767786 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:27.767759    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d4d0a3c-0312-4063-b331-3735c26dcc4c-plugins-config-volume" pod "8d4d0a3c-0312-4063-b331-3735c26dcc4c" (UID: "8d4d0a3c-0312-4063-b331-3735c26dcc4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:58:27.768277 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:27.768252    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d4d0a3c-0312-4063-b331-3735c26dcc4c-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:27.769592 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:27.769568    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4d0a3c-0312-4063-b331-3735c26dcc4c-kube-api-access-hwk89" pod "8d4d0a3c-0312-4063-b331-3735c26dcc4c" (UID: "8d4d0a3c-0312-4063-b331-3735c26dcc4c"). InnerVolumeSpecName "kube-api-access-hwk89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:58:27.870227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:27.870172    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwk89\" (UniqueName: \"kubernetes.io/projected/8d4d0a3c-0312-4063-b331-3735c26dcc4c-kube-api-access-hwk89\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:27.949129 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:27.949084    5591 patch_prober.go:29] interesting pod/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 19:58:27.949346 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:27.949179    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq" podUID="f226a9f7-2809-4890-8cc3-29013e49d774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 19:58:27.961041 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:27.961008    5591 generic.go:356] "Generic (PLEG): container finished" podID="8d4d0a3c-0312-4063-b331-3735c26dcc4c" containerID="a6c8fc09e8dd7b17f9387400c7949ad52f2d61be1ad8cad715c1d7316dd424aa" exitCode=0
Nov 03 19:58:27.961262 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:27.961077    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr"
Nov 03 19:58:27.961262 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:27.961093    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr" event={"ID":"8d4d0a3c-0312-4063-b331-3735c26dcc4c","Type":"ContainerDied","Data":"a6c8fc09e8dd7b17f9387400c7949ad52f2d61be1ad8cad715c1d7316dd424aa"}
Nov 03 19:58:27.961262 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:27.961184    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr" event={"ID":"8d4d0a3c-0312-4063-b331-3735c26dcc4c","Type":"ContainerDied","Data":"0f11e166e185cfd6c1b5599a5eef6f5a4c5b011e693eec7ce82f39d499db6a21"}
Nov 03 19:58:27.961262 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:27.961236    5591 scope.go:122] "RemoveContainer" containerID="a6c8fc09e8dd7b17f9387400c7949ad52f2d61be1ad8cad715c1d7316dd424aa"
Nov 03 19:58:27.962979 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:27.962953    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr" event={"ID":"a08831b1-e194-4e39-b23d-6232d8d9851e","Type":"ContainerStarted","Data":"3cae4e3e2789caad79b9137981f7c949c98f13c5efb531f0534c6c59454011f6"}
Nov 03 19:58:27.963102 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:27.963004    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr" event={"ID":"a08831b1-e194-4e39-b23d-6232d8d9851e","Type":"ContainerStarted","Data":"8e4bbbbe1d8c6064741e6eb9cfd7122a6726052a54b2e773a023f2567c1c019d"}
Nov 03 19:58:27.973159 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:27.973138    5591 scope.go:122] "RemoveContainer" containerID="a6c8fc09e8dd7b17f9387400c7949ad52f2d61be1ad8cad715c1d7316dd424aa"
Nov 03 19:58:27.973568 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:58:27.973544    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6c8fc09e8dd7b17f9387400c7949ad52f2d61be1ad8cad715c1d7316dd424aa\": container with ID starting with a6c8fc09e8dd7b17f9387400c7949ad52f2d61be1ad8cad715c1d7316dd424aa not found: ID does not exist" containerID="a6c8fc09e8dd7b17f9387400c7949ad52f2d61be1ad8cad715c1d7316dd424aa"
Nov 03 19:58:27.973643 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:27.973578    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6c8fc09e8dd7b17f9387400c7949ad52f2d61be1ad8cad715c1d7316dd424aa"} err="failed to get container status \"a6c8fc09e8dd7b17f9387400c7949ad52f2d61be1ad8cad715c1d7316dd424aa\": rpc error: code = NotFound desc = could not find container \"a6c8fc09e8dd7b17f9387400c7949ad52f2d61be1ad8cad715c1d7316dd424aa\": container with ID starting with a6c8fc09e8dd7b17f9387400c7949ad52f2d61be1ad8cad715c1d7316dd424aa not found: ID does not exist"
Nov 03 19:58:28.048470 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:28.047496    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr" podStartSLOduration=3.047477835 podStartE2EDuration="3.047477835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:28.012587731 +0000 UTC m=+1256.784880751" watchObservedRunningTime="2026-11-03 19:58:28.047477835 +0000 UTC m=+1256.819770853"
Nov 03 19:58:28.048870 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:28.048839    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr"]
Nov 03 19:58:28.052325 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:28.052300    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6a7dc9-666cdfb855-rdxgr"]
Nov 03 19:58:28.245099 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:28.245069    5591 patch_prober.go:29] interesting pod/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 19:58:28.245773 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:28.245744    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx" podUID="41a51c01-3bd6-4414-9f7e-6347d2d59727" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 19:58:28.305289 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:28.303135    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:28.305289 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:28.303296    5591 patch_prober.go:29] interesting pod/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 03 19:58:28.305289 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:28.303352    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr" podUID="a08831b1-e194-4e39-b23d-6232d8d9851e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 03 19:58:28.767089 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:58:28.767057    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:58:29.067094 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:29.067003    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:58:29.068504 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:29.068478    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:58:29.069196 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:29.069177    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 19:58:29.249582 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:29.249538    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:58:29.250966 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:29.250941    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:58:29.331279 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:29.329053    5591 patch_prober.go:29] interesting pod/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 03 19:58:29.331279 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:29.329125    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr" podUID="a08831b1-e194-4e39-b23d-6232d8d9851e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 03 19:58:29.466244 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:29.465558    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d4d0a3c-0312-4063-b331-3735c26dcc4c" path="/var/lib/kubelet/pods/8d4d0a3c-0312-4063-b331-3735c26dcc4c/volumes"
Nov 03 19:58:30.299713 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:30.299675    5591 patch_prober.go:29] interesting pod/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 03 19:58:30.300331 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:30.300297    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr" podUID="a08831b1-e194-4e39-b23d-6232d8d9851e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 03 19:58:30.659415 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:30.659324    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:58:31.303411 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:31.303370    5591 patch_prober.go:29] interesting pod/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 03 19:58:31.306804 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:31.306769    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr" podUID="a08831b1-e194-4e39-b23d-6232d8d9851e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 03 19:58:32.298231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:32.297999    5591 patch_prober.go:29] interesting pod/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 03 19:58:32.298231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:32.298062    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr" podUID="a08831b1-e194-4e39-b23d-6232d8d9851e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 03 19:58:32.622274 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:32.622180    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb"]
Nov 03 19:58:32.622728 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:32.622427    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb" podUID="3ce9e626-faa5-4b01-8849-971bc8def06c" containerName="maas-api" containerID="cri-o://d3dce1d5133bd8f4520c79a3c89850fe761a89ee66847e92663879131b39c5c5" gracePeriod=30
Nov 03 19:58:33.379228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:33.374888    5591 patch_prober.go:29] interesting pod/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 03 19:58:33.379228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:33.374954    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr" podUID="a08831b1-e194-4e39-b23d-6232d8d9851e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 03 19:58:33.417098 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:33.416857    5591 generic.go:356] "Generic (PLEG): container finished" podID="3ce9e626-faa5-4b01-8849-971bc8def06c" containerID="d3dce1d5133bd8f4520c79a3c89850fe761a89ee66847e92663879131b39c5c5" exitCode=0
Nov 03 19:58:33.418121 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:33.418015    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb" event={"ID":"3ce9e626-faa5-4b01-8849-971bc8def06c","Type":"ContainerDied","Data":"d3dce1d5133bd8f4520c79a3c89850fe761a89ee66847e92663879131b39c5c5"}
Nov 03 19:58:33.418617 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:33.418598    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb"
Nov 03 19:58:33.493411 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:33.493372    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ce9e626-faa5-4b01-8849-971bc8def06c-kube-api-access-x626c\" (UniqueName: \"kubernetes.io/projected/3ce9e626-faa5-4b01-8849-971bc8def06c-kube-api-access-x626c\") pod \"3ce9e626-faa5-4b01-8849-971bc8def06c\" (UID: \"3ce9e626-faa5-4b01-8849-971bc8def06c\") "
Nov 03 19:58:33.493910 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:33.493887    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ce9e626-faa5-4b01-8849-971bc8def06c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ce9e626-faa5-4b01-8849-971bc8def06c-maas-api-tls\") pod \"3ce9e626-faa5-4b01-8849-971bc8def06c\" (UID: \"3ce9e626-faa5-4b01-8849-971bc8def06c\") "
Nov 03 19:58:33.494105 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:33.494088    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ce9e626-faa5-4b01-8849-971bc8def06c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce9e626-faa5-4b01-8849-971bc8def06c-openshift-service-ca\") pod \"3ce9e626-faa5-4b01-8849-971bc8def06c\" (UID: \"3ce9e626-faa5-4b01-8849-971bc8def06c\") "
Nov 03 19:58:33.494281 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:33.494260    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ce9e626-faa5-4b01-8849-971bc8def06c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ce9e626-faa5-4b01-8849-971bc8def06c-metrics-certs\") pod \"3ce9e626-faa5-4b01-8849-971bc8def06c\" (UID: \"3ce9e626-faa5-4b01-8849-971bc8def06c\") "
Nov 03 19:58:33.502392 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:33.502364    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ce9e626-faa5-4b01-8849-971bc8def06c-openshift-service-ca" pod "3ce9e626-faa5-4b01-8849-971bc8def06c" (UID: "3ce9e626-faa5-4b01-8849-971bc8def06c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:58:33.505019 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:33.504993    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ce9e626-faa5-4b01-8849-971bc8def06c-metrics-certs" pod "3ce9e626-faa5-4b01-8849-971bc8def06c" (UID: "3ce9e626-faa5-4b01-8849-971bc8def06c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:58:33.505326 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:33.505281    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce9e626-faa5-4b01-8849-971bc8def06c-kube-api-access-x626c" pod "3ce9e626-faa5-4b01-8849-971bc8def06c" (UID: "3ce9e626-faa5-4b01-8849-971bc8def06c"). InnerVolumeSpecName "kube-api-access-x626c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:58:33.506352 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:33.506327    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ce9e626-faa5-4b01-8849-971bc8def06c-maas-api-tls" pod "3ce9e626-faa5-4b01-8849-971bc8def06c" (UID: "3ce9e626-faa5-4b01-8849-971bc8def06c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:58:33.603635 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:33.603599    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ce9e626-faa5-4b01-8849-971bc8def06c-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:33.603886 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:33.603865    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ce9e626-faa5-4b01-8849-971bc8def06c-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:33.604006 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:33.603992    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ce9e626-faa5-4b01-8849-971bc8def06c-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:33.604163 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:33.604145    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x626c\" (UniqueName: \"kubernetes.io/projected/3ce9e626-faa5-4b01-8849-971bc8def06c-kube-api-access-x626c\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:34.296126 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:34.296089    5591 patch_prober.go:29] interesting pod/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 03 19:58:34.296764 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:34.296720    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr" podUID="a08831b1-e194-4e39-b23d-6232d8d9851e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 03 19:58:34.555930 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:34.555612    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb"
Nov 03 19:58:34.555930 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:34.555645    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb" event={"ID":"3ce9e626-faa5-4b01-8849-971bc8def06c","Type":"ContainerDied","Data":"58892f7915d1785b67a58285bc71c2aa057c2cfc41bf363e4f47c23f18f863ec"}
Nov 03 19:58:34.555930 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:34.555725    5591 scope.go:122] "RemoveContainer" containerID="d3dce1d5133bd8f4520c79a3c89850fe761a89ee66847e92663879131b39c5c5"
Nov 03 19:58:34.616456 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:34.616411    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb"]
Nov 03 19:58:34.619821 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:34.619785    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6a7dc9-89d67655b-bmlcb"]
Nov 03 19:58:35.353917 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:35.353875    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ce9e626-faa5-4b01-8849-971bc8def06c" path="/var/lib/kubelet/pods/3ce9e626-faa5-4b01-8849-971bc8def06c/volumes"
Nov 03 19:58:35.484489 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:35.484462    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:35.485805 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:35.485775    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:35.486838 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:35.486814    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 19:58:38.713295 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:38.713259    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p"]
Nov 03 19:58:38.713948 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:38.713530    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p" podUID="ac0550ab-2fdd-41f1-90d9-081287e63228" containerName="maas-api" containerID="cri-o://0723aadd93c8327e817ae8d96601b72b9c4cdc0145807df980f2fc8352f714ff" gracePeriod=30
Nov 03 19:58:38.820556 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:38.820521    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7"]
Nov 03 19:58:38.821065 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:38.821027    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7" podUID="0d82d43e-048c-4e90-968b-a86c91ca38a2" containerName="payload-processing" containerID="cri-o://e5a9eaf5cdd5dd786235cfd73d0c3409bb186b02a25d575e22a8b0cf1c7ef866" gracePeriod=30
Nov 03 19:58:39.356056 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.355732    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p"
Nov 03 19:58:39.460055 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.459986    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac0550ab-2fdd-41f1-90d9-081287e63228-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac0550ab-2fdd-41f1-90d9-081287e63228-maas-api-tls\") pod \"ac0550ab-2fdd-41f1-90d9-081287e63228\" (UID: \"ac0550ab-2fdd-41f1-90d9-081287e63228\") "
Nov 03 19:58:39.460276 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.460075    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac0550ab-2fdd-41f1-90d9-081287e63228-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0550ab-2fdd-41f1-90d9-081287e63228-openshift-service-ca\") pod \"ac0550ab-2fdd-41f1-90d9-081287e63228\" (UID: \"ac0550ab-2fdd-41f1-90d9-081287e63228\") "
Nov 03 19:58:39.460357 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.460315    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac0550ab-2fdd-41f1-90d9-081287e63228-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac0550ab-2fdd-41f1-90d9-081287e63228-metrics-certs\") pod \"ac0550ab-2fdd-41f1-90d9-081287e63228\" (UID: \"ac0550ab-2fdd-41f1-90d9-081287e63228\") "
Nov 03 19:58:39.460419 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.460399    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac0550ab-2fdd-41f1-90d9-081287e63228-kube-api-access-prqch\" (UniqueName: \"kubernetes.io/projected/ac0550ab-2fdd-41f1-90d9-081287e63228-kube-api-access-prqch\") pod \"ac0550ab-2fdd-41f1-90d9-081287e63228\" (UID: \"ac0550ab-2fdd-41f1-90d9-081287e63228\") "
Nov 03 19:58:39.462071 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.462046    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac0550ab-2fdd-41f1-90d9-081287e63228-openshift-service-ca" pod "ac0550ab-2fdd-41f1-90d9-081287e63228" (UID: "ac0550ab-2fdd-41f1-90d9-081287e63228"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:58:39.486408 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.486373    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac0550ab-2fdd-41f1-90d9-081287e63228-maas-api-tls" pod "ac0550ab-2fdd-41f1-90d9-081287e63228" (UID: "ac0550ab-2fdd-41f1-90d9-081287e63228"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:58:39.486905 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.486856    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac0550ab-2fdd-41f1-90d9-081287e63228-metrics-certs" pod "ac0550ab-2fdd-41f1-90d9-081287e63228" (UID: "ac0550ab-2fdd-41f1-90d9-081287e63228"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:58:39.490738 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.490639    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac0550ab-2fdd-41f1-90d9-081287e63228-kube-api-access-prqch" pod "ac0550ab-2fdd-41f1-90d9-081287e63228" (UID: "ac0550ab-2fdd-41f1-90d9-081287e63228"). InnerVolumeSpecName "kube-api-access-prqch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:58:39.500190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.500104    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7"
Nov 03 19:58:39.563478 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.563443    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d82d43e-048c-4e90-968b-a86c91ca38a2-kube-api-access-vntqm\" (UniqueName: \"kubernetes.io/projected/0d82d43e-048c-4e90-968b-a86c91ca38a2-kube-api-access-vntqm\") pod \"0d82d43e-048c-4e90-968b-a86c91ca38a2\" (UID: \"0d82d43e-048c-4e90-968b-a86c91ca38a2\") "
Nov 03 19:58:39.563733 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.563716    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d82d43e-048c-4e90-968b-a86c91ca38a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d82d43e-048c-4e90-968b-a86c91ca38a2-plugins-config-volume\") pod \"0d82d43e-048c-4e90-968b-a86c91ca38a2\" (UID: \"0d82d43e-048c-4e90-968b-a86c91ca38a2\") "
Nov 03 19:58:39.564364 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.564336    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac0550ab-2fdd-41f1-90d9-081287e63228-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:39.564489 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.564372    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prqch\" (UniqueName: \"kubernetes.io/projected/ac0550ab-2fdd-41f1-90d9-081287e63228-kube-api-access-prqch\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:39.564489 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.564393    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac0550ab-2fdd-41f1-90d9-081287e63228-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:39.564489 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.564410    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac0550ab-2fdd-41f1-90d9-081287e63228-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:39.564812 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.564783    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d82d43e-048c-4e90-968b-a86c91ca38a2-plugins-config-volume" pod "0d82d43e-048c-4e90-968b-a86c91ca38a2" (UID: "0d82d43e-048c-4e90-968b-a86c91ca38a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:58:39.592232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.579924    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d82d43e-048c-4e90-968b-a86c91ca38a2-kube-api-access-vntqm" pod "0d82d43e-048c-4e90-968b-a86c91ca38a2" (UID: "0d82d43e-048c-4e90-968b-a86c91ca38a2"). InnerVolumeSpecName "kube-api-access-vntqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:58:39.672307 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.672195    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vntqm\" (UniqueName: \"kubernetes.io/projected/0d82d43e-048c-4e90-968b-a86c91ca38a2-kube-api-access-vntqm\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:39.672603 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.672534    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d82d43e-048c-4e90-968b-a86c91ca38a2-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:39.837103 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.837071    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq"]
Nov 03 19:58:39.838112 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.838093    5591 cpu_manager.go:414] "containerMap: removing container" podUID="0d82d43e-048c-4e90-968b-a86c91ca38a2" containerName="payload-processing"
Nov 03 19:58:39.838112 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.838113    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d82d43e-048c-4e90-968b-a86c91ca38a2" containerName="payload-processing"
Nov 03 19:58:39.838276 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.838146    5591 cpu_manager.go:414] "containerMap: removing container" podUID="8d4d0a3c-0312-4063-b331-3735c26dcc4c" containerName="payload-pre-processing"
Nov 03 19:58:39.838276 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.838152    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d4d0a3c-0312-4063-b331-3735c26dcc4c" containerName="payload-pre-processing"
Nov 03 19:58:39.838276 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.838164    5591 cpu_manager.go:414] "containerMap: removing container" podUID="3ce9e626-faa5-4b01-8849-971bc8def06c" containerName="maas-api"
Nov 03 19:58:39.838276 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.838169    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ce9e626-faa5-4b01-8849-971bc8def06c" containerName="maas-api"
Nov 03 19:58:39.838276 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.838187    5591 cpu_manager.go:414] "containerMap: removing container" podUID="ac0550ab-2fdd-41f1-90d9-081287e63228" containerName="maas-api"
Nov 03 19:58:39.838276 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.838191    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac0550ab-2fdd-41f1-90d9-081287e63228" containerName="maas-api"
Nov 03 19:58:39.838458 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.838304    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d82d43e-048c-4e90-968b-a86c91ca38a2" containerName="payload-processing"
Nov 03 19:58:39.838458 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.838316    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d4d0a3c-0312-4063-b331-3735c26dcc4c" containerName="payload-pre-processing"
Nov 03 19:58:39.838458 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.838324    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ce9e626-faa5-4b01-8849-971bc8def06c" containerName="maas-api"
Nov 03 19:58:39.838458 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.838332    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac0550ab-2fdd-41f1-90d9-081287e63228" containerName="maas-api"
Nov 03 19:58:39.840974 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.840956    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq"
Nov 03 19:58:39.841936 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.841910    5591 generic.go:356] "Generic (PLEG): container finished" podID="ac0550ab-2fdd-41f1-90d9-081287e63228" containerID="0723aadd93c8327e817ae8d96601b72b9c4cdc0145807df980f2fc8352f714ff" exitCode=0
Nov 03 19:58:39.842102 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.841966    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p"
Nov 03 19:58:39.842102 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.842000    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p" event={"ID":"ac0550ab-2fdd-41f1-90d9-081287e63228","Type":"ContainerDied","Data":"0723aadd93c8327e817ae8d96601b72b9c4cdc0145807df980f2fc8352f714ff"}
Nov 03 19:58:39.842400 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.842137    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p" event={"ID":"ac0550ab-2fdd-41f1-90d9-081287e63228","Type":"ContainerDied","Data":"75e54eb831d28b4542bf171615565bcd3b67ee5ca6af3d5e7edffdd8c1dad308"}
Nov 03 19:58:39.842400 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.842178    5591 scope.go:122] "RemoveContainer" containerID="0723aadd93c8327e817ae8d96601b72b9c4cdc0145807df980f2fc8352f714ff"
Nov 03 19:58:39.849936 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.847520    5591 generic.go:356] "Generic (PLEG): container finished" podID="0d82d43e-048c-4e90-968b-a86c91ca38a2" containerID="e5a9eaf5cdd5dd786235cfd73d0c3409bb186b02a25d575e22a8b0cf1c7ef866" exitCode=0
Nov 03 19:58:39.849936 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.847586    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7" event={"ID":"0d82d43e-048c-4e90-968b-a86c91ca38a2","Type":"ContainerDied","Data":"e5a9eaf5cdd5dd786235cfd73d0c3409bb186b02a25d575e22a8b0cf1c7ef866"}
Nov 03 19:58:39.849936 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.847626    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7" event={"ID":"0d82d43e-048c-4e90-968b-a86c91ca38a2","Type":"ContainerDied","Data":"106f63835eaee38a7fb3c22e72f020c5847a0721f19f0da2e9c35ff74ae898d6"}
Nov 03 19:58:39.849936 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.847707    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7"
Nov 03 19:58:39.856916 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.856891    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq"]
Nov 03 19:58:39.857876 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.857855    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-f493f7\""
Nov 03 19:58:39.862054 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.862031    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c"]
Nov 03 19:58:39.864641 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.864503    5591 scope.go:122] "RemoveContainer" containerID="0723aadd93c8327e817ae8d96601b72b9c4cdc0145807df980f2fc8352f714ff"
Nov 03 19:58:39.865069 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:58:39.865047    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0723aadd93c8327e817ae8d96601b72b9c4cdc0145807df980f2fc8352f714ff\": container with ID starting with 0723aadd93c8327e817ae8d96601b72b9c4cdc0145807df980f2fc8352f714ff not found: ID does not exist" containerID="0723aadd93c8327e817ae8d96601b72b9c4cdc0145807df980f2fc8352f714ff"
Nov 03 19:58:39.865179 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.865079    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0723aadd93c8327e817ae8d96601b72b9c4cdc0145807df980f2fc8352f714ff"} err="failed to get container status \"0723aadd93c8327e817ae8d96601b72b9c4cdc0145807df980f2fc8352f714ff\": rpc error: code = NotFound desc = could not find container \"0723aadd93c8327e817ae8d96601b72b9c4cdc0145807df980f2fc8352f714ff\": container with ID starting with 0723aadd93c8327e817ae8d96601b72b9c4cdc0145807df980f2fc8352f714ff not found: ID does not exist"
Nov 03 19:58:39.865179 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.865102    5591 scope.go:122] "RemoveContainer" containerID="e5a9eaf5cdd5dd786235cfd73d0c3409bb186b02a25d575e22a8b0cf1c7ef866"
Nov 03 19:58:39.867021 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.867000    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c"
Nov 03 19:58:39.874197 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.874031    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c"]
Nov 03 19:58:39.879161 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.879138    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ntn2\" (UniqueName: \"kubernetes.io/projected/7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2-kube-api-access-9ntn2\") pod \"payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c\" (UID: \"7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c"
Nov 03 19:58:39.879363 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.879343    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c2a580-9312-4873-8b6d-f8970cab6e18-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq\" (UID: \"d9c2a580-9312-4873-8b6d-f8970cab6e18\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq"
Nov 03 19:58:39.879550 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.879535    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqw7v\" (UniqueName: \"kubernetes.io/projected/d9c2a580-9312-4873-8b6d-f8970cab6e18-kube-api-access-lqw7v\") pod \"payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq\" (UID: \"d9c2a580-9312-4873-8b6d-f8970cab6e18\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq"
Nov 03 19:58:39.879772 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.879748    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c\" (UID: \"7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c"
Nov 03 19:58:39.886297 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.882535    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-f493f7-dockercfg-pjtp5\""
Nov 03 19:58:39.896255 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.895090    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p"]
Nov 03 19:58:39.899890 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.899866    5591 scope.go:122] "RemoveContainer" containerID="e5a9eaf5cdd5dd786235cfd73d0c3409bb186b02a25d575e22a8b0cf1c7ef866"
Nov 03 19:58:39.900310 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:58:39.900286    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a9eaf5cdd5dd786235cfd73d0c3409bb186b02a25d575e22a8b0cf1c7ef866\": container with ID starting with e5a9eaf5cdd5dd786235cfd73d0c3409bb186b02a25d575e22a8b0cf1c7ef866 not found: ID does not exist" containerID="e5a9eaf5cdd5dd786235cfd73d0c3409bb186b02a25d575e22a8b0cf1c7ef866"
Nov 03 19:58:39.900397 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.900318    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a9eaf5cdd5dd786235cfd73d0c3409bb186b02a25d575e22a8b0cf1c7ef866"} err="failed to get container status \"e5a9eaf5cdd5dd786235cfd73d0c3409bb186b02a25d575e22a8b0cf1c7ef866\": rpc error: code = NotFound desc = could not find container \"e5a9eaf5cdd5dd786235cfd73d0c3409bb186b02a25d575e22a8b0cf1c7ef866\": container with ID starting with e5a9eaf5cdd5dd786235cfd73d0c3409bb186b02a25d575e22a8b0cf1c7ef866 not found: ID does not exist"
Nov 03 19:58:39.900862 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.900841    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p"]
Nov 03 19:58:39.910783 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.910761    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7"]
Nov 03 19:58:39.917966 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:39.917941    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9b3619-57786cf4cb-hgkw7"]
Nov 03 19:58:40.040302 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.037774    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ntn2\" (UniqueName: \"kubernetes.io/projected/7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2-kube-api-access-9ntn2\") pod \"payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c\" (UID: \"7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c"
Nov 03 19:58:40.040302 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.037835    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c2a580-9312-4873-8b6d-f8970cab6e18-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq\" (UID: \"d9c2a580-9312-4873-8b6d-f8970cab6e18\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq"
Nov 03 19:58:40.040302 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.037930    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqw7v\" (UniqueName: \"kubernetes.io/projected/d9c2a580-9312-4873-8b6d-f8970cab6e18-kube-api-access-lqw7v\") pod \"payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq\" (UID: \"d9c2a580-9312-4873-8b6d-f8970cab6e18\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq"
Nov 03 19:58:40.040302 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.037971    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c\" (UID: \"7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c"
Nov 03 19:58:40.040302 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.039686    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c2a580-9312-4873-8b6d-f8970cab6e18-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq\" (UID: \"d9c2a580-9312-4873-8b6d-f8970cab6e18\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq"
Nov 03 19:58:40.099244 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.089745    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c\" (UID: \"7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c"
Nov 03 19:58:40.100383 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.100312    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ntn2\" (UniqueName: \"kubernetes.io/projected/7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2-kube-api-access-9ntn2\") pod \"payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c\" (UID: \"7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c"
Nov 03 19:58:40.123226 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.117978    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqw7v\" (UniqueName: \"kubernetes.io/projected/d9c2a580-9312-4873-8b6d-f8970cab6e18-kube-api-access-lqw7v\") pod \"payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq\" (UID: \"d9c2a580-9312-4873-8b6d-f8970cab6e18\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq"
Nov 03 19:58:40.165107 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.165032    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq"
Nov 03 19:58:40.201326 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.201041    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c"
Nov 03 19:58:40.612635 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.612605    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq"]
Nov 03 19:58:40.613340 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:58:40.613311    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c2a580_9312_4873_8b6d_f8970cab6e18.slice/crio-ceb00877cb11eccdb33607b39d3e89b424bd08eb61c0b00a5dac43ed9ebf8f9e WatchSource:0}: Error finding container ceb00877cb11eccdb33607b39d3e89b424bd08eb61c0b00a5dac43ed9ebf8f9e: Status 404 returned error can't find the container with id ceb00877cb11eccdb33607b39d3e89b424bd08eb61c0b00a5dac43ed9ebf8f9e
Nov 03 19:58:40.638151 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.638128    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c"]
Nov 03 19:58:40.640702 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:58:40.640656    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e13f9b7_9144_4bc4_9ce8_5a9d3d3641b2.slice/crio-02c4a500eef3187c74d89f0bfcb1b5c16c4644767bef991fd05f4209e2f12586 WatchSource:0}: Error finding container 02c4a500eef3187c74d89f0bfcb1b5c16c4644767bef991fd05f4209e2f12586: Status 404 returned error can't find the container with id 02c4a500eef3187c74d89f0bfcb1b5c16c4644767bef991fd05f4209e2f12586
Nov 03 19:58:40.857993 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.857954    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c" event={"ID":"7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2","Type":"ContainerStarted","Data":"2a889f04506b841873ec3c39c56a9370004baa0db49b8407088e49f46a878c2c"}
Nov 03 19:58:40.858604 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.858030    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c"
Nov 03 19:58:40.858604 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.858057    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c" event={"ID":"7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2","Type":"ContainerStarted","Data":"02c4a500eef3187c74d89f0bfcb1b5c16c4644767bef991fd05f4209e2f12586"}
Nov 03 19:58:40.861774 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.861692    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq" event={"ID":"d9c2a580-9312-4873-8b6d-f8970cab6e18","Type":"ContainerStarted","Data":"94eac53a7af7d417f6b5c3a47cce639c12b849302653293dc6c516ce2e213365"}
Nov 03 19:58:40.861774 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.861745    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq" event={"ID":"d9c2a580-9312-4873-8b6d-f8970cab6e18","Type":"ContainerStarted","Data":"ceb00877cb11eccdb33607b39d3e89b424bd08eb61c0b00a5dac43ed9ebf8f9e"}
Nov 03 19:58:40.862104 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.862085    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq"
Nov 03 19:58:40.865191 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.865170    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh"]
Nov 03 19:58:40.868855 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.868839    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh"
Nov 03 19:58:40.875784 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.875641    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-277dc9\""
Nov 03 19:58:40.891611 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.890588    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh"]
Nov 03 19:58:40.894864 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.894803    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c" podStartSLOduration=1.8947831069999999 podStartE2EDuration="1.894783107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:40.883513116 +0000 UTC m=+1269.655806135" watchObservedRunningTime="2026-11-03 19:58:40.894783107 +0000 UTC m=+1269.667076125"
Nov 03 19:58:40.900710 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.900674    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4"]
Nov 03 19:58:40.905127 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.905105    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4"
Nov 03 19:58:40.923560 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.923529    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4"]
Nov 03 19:58:40.923866 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.923751    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-277dc9-dockercfg-shzdv\""
Nov 03 19:58:40.927467 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.927405    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq" podStartSLOduration=1.92738452 podStartE2EDuration="1.92738452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:40.920706382 +0000 UTC m=+1269.692999401" watchObservedRunningTime="2026-11-03 19:58:40.92738452 +0000 UTC m=+1269.699677539"
Nov 03 19:58:40.968461 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.968424    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db5bfc00-f082-405e-8d43-843cbaf95b52-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh\" (UID: \"db5bfc00-f082-405e-8d43-843cbaf95b52\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh"
Nov 03 19:58:40.968666 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:40.968576    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lvch\" (UniqueName: \"kubernetes.io/projected/db5bfc00-f082-405e-8d43-843cbaf95b52-kube-api-access-8lvch\") pod \"payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh\" (UID: \"db5bfc00-f082-405e-8d43-843cbaf95b52\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh"
Nov 03 19:58:41.080234 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:41.071980    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvt77\" (UniqueName: \"kubernetes.io/projected/309d5f96-0577-45a6-8855-727412314a51-kube-api-access-tvt77\") pod \"payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4\" (UID: \"309d5f96-0577-45a6-8855-727412314a51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4"
Nov 03 19:58:41.080234 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:41.072082    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/309d5f96-0577-45a6-8855-727412314a51-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4\" (UID: \"309d5f96-0577-45a6-8855-727412314a51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4"
Nov 03 19:58:41.080234 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:41.072131    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db5bfc00-f082-405e-8d43-843cbaf95b52-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh\" (UID: \"db5bfc00-f082-405e-8d43-843cbaf95b52\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh"
Nov 03 19:58:41.080234 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:41.072256    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lvch\" (UniqueName: \"kubernetes.io/projected/db5bfc00-f082-405e-8d43-843cbaf95b52-kube-api-access-8lvch\") pod \"payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh\" (UID: \"db5bfc00-f082-405e-8d43-843cbaf95b52\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh"
Nov 03 19:58:41.080234 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:41.079332    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db5bfc00-f082-405e-8d43-843cbaf95b52-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh\" (UID: \"db5bfc00-f082-405e-8d43-843cbaf95b52\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh"
Nov 03 19:58:41.142292 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:41.136679    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lvch\" (UniqueName: \"kubernetes.io/projected/db5bfc00-f082-405e-8d43-843cbaf95b52-kube-api-access-8lvch\") pod \"payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh\" (UID: \"db5bfc00-f082-405e-8d43-843cbaf95b52\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh"
Nov 03 19:58:41.188243 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:41.185701    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/309d5f96-0577-45a6-8855-727412314a51-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4\" (UID: \"309d5f96-0577-45a6-8855-727412314a51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4"
Nov 03 19:58:41.188243 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:41.186059    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvt77\" (UniqueName: \"kubernetes.io/projected/309d5f96-0577-45a6-8855-727412314a51-kube-api-access-tvt77\") pod \"payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4\" (UID: \"309d5f96-0577-45a6-8855-727412314a51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4"
Nov 03 19:58:41.188243 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:41.187168    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/309d5f96-0577-45a6-8855-727412314a51-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4\" (UID: \"309d5f96-0577-45a6-8855-727412314a51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4"
Nov 03 19:58:41.196229 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:41.194450    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh"
Nov 03 19:58:41.214135 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:41.214102    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvt77\" (UniqueName: \"kubernetes.io/projected/309d5f96-0577-45a6-8855-727412314a51-kube-api-access-tvt77\") pod \"payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4\" (UID: \"309d5f96-0577-45a6-8855-727412314a51\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4"
Nov 03 19:58:41.237468 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:41.234985    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4"
Nov 03 19:58:41.558670 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:41.558567    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d82d43e-048c-4e90-968b-a86c91ca38a2" path="/var/lib/kubelet/pods/0d82d43e-048c-4e90-968b-a86c91ca38a2/volumes"
Nov 03 19:58:41.559754 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:41.559726    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac0550ab-2fdd-41f1-90d9-081287e63228" path="/var/lib/kubelet/pods/ac0550ab-2fdd-41f1-90d9-081287e63228/volumes"
Nov 03 19:58:41.799748 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:41.799716    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4"]
Nov 03 19:58:41.802122 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:58:41.802088    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod309d5f96_0577_45a6_8855_727412314a51.slice/crio-0fa3554d6c21f7a8eca61e9201bd86c82537097c6f587eea2a85808d82b78bc1 WatchSource:0}: Error finding container 0fa3554d6c21f7a8eca61e9201bd86c82537097c6f587eea2a85808d82b78bc1: Status 404 returned error can't find the container with id 0fa3554d6c21f7a8eca61e9201bd86c82537097c6f587eea2a85808d82b78bc1
Nov 03 19:58:41.840165 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:41.840139    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh"]
Nov 03 19:58:41.873518 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:41.873492    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4" event={"ID":"309d5f96-0577-45a6-8855-727412314a51","Type":"ContainerStarted","Data":"0fa3554d6c21f7a8eca61e9201bd86c82537097c6f587eea2a85808d82b78bc1"}
Nov 03 19:58:41.874884 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:41.874857    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh" event={"ID":"db5bfc00-f082-405e-8d43-843cbaf95b52","Type":"ContainerStarted","Data":"c2deafa393be9fbbcfb0cee63b2b780bfa982cd1e57d335d394bf51198d3667b"}
Nov 03 19:58:42.167112 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:42.165310    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9b3619-58b74894fb-vcj7p" podUID="ac0550ab-2fdd-41f1-90d9-081287e63228" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.117:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:58:42.884320 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:42.884269    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4" event={"ID":"309d5f96-0577-45a6-8855-727412314a51","Type":"ContainerStarted","Data":"53bf3186af373d21a94c9eebf3a15b8964c59391ca81a814976d5f734b895c7a"}
Nov 03 19:58:42.884826 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:42.884367    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4"
Nov 03 19:58:42.886190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:42.886167    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh" event={"ID":"db5bfc00-f082-405e-8d43-843cbaf95b52","Type":"ContainerStarted","Data":"62f7076414d6d48e66c1196cca0beddf60c3285d705c1e48dec5cf9e17b4c7c2"}
Nov 03 19:58:42.886356 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:42.886286    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh"
Nov 03 19:58:42.907049 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:42.906999    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4" podStartSLOduration=2.906983924 podStartE2EDuration="2.906983924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:42.902312549 +0000 UTC m=+1271.674605567" watchObservedRunningTime="2026-11-03 19:58:42.906983924 +0000 UTC m=+1271.679276979"
Nov 03 19:58:42.922059 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:42.921998    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh" podStartSLOduration=2.92198366 podStartE2EDuration="2.92198366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:42.919988445 +0000 UTC m=+1271.692281481" watchObservedRunningTime="2026-11-03 19:58:42.92198366 +0000 UTC m=+1271.694276677"
Nov 03 19:58:45.185600 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:45.185091    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"]
Nov 03 19:58:45.186636 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:45.186597    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx" podUID="01b87ad5-beef-41b7-99be-693434e57bbc" containerName="istio-proxy" containerID="cri-o://7fa297fc2046c52f70dd6a0e4b6cc424e082d815df3ee0d96a1a40cd628d33bf" gracePeriod=30
Nov 03 19:58:45.189788 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:45.189762    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"]
Nov 03 19:58:45.190061 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:45.190032    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7" podUID="2e0b0d0d-df65-4dfe-9333-1237cb41b8a4" containerName="istio-proxy" containerID="cri-o://f451029784550ce6a6afc7134192e318ea92e5df08775922488f18a44b4363d0" gracePeriod=30
Nov 03 19:58:46.156893 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:46.156855    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:58:46.157060 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:46.156928    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx" podUID="01b87ad5-beef-41b7-99be-693434e57bbc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:58:50.587246 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.587085    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:58:50.588388 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.588315    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:58:50.706077 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.706045    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-kube-api-access-8wkjl\" (UniqueName: \"kubernetes.io/projected/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-kube-api-access-8wkjl\") pod \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") "
Nov 03 19:58:50.706376 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.706095    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-workload-certs\") pod \"01b87ad5-beef-41b7-99be-693434e57bbc\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") "
Nov 03 19:58:50.706376 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.706127    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-workload-socket\") pod \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") "
Nov 03 19:58:50.706376 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.706157    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-envoy\") pod \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") "
Nov 03 19:58:50.706376 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.706198    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-data\") pod \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") "
Nov 03 19:58:50.706376 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.706334    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01b87ad5-beef-41b7-99be-693434e57bbc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b87ad5-beef-41b7-99be-693434e57bbc-istio-podinfo\") pod \"01b87ad5-beef-41b7-99be-693434e57bbc\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") "
Nov 03 19:58:50.706376 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.706372    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istiod-ca-cert\") pod \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") "
Nov 03 19:58:50.706773 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.706475    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-workload-certs" pod "01b87ad5-beef-41b7-99be-693434e57bbc" (UID: "01b87ad5-beef-41b7-99be-693434e57bbc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:50.706773 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.706503    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01b87ad5-beef-41b7-99be-693434e57bbc-kube-api-access-z8spg\" (UniqueName: \"kubernetes.io/projected/01b87ad5-beef-41b7-99be-693434e57bbc-kube-api-access-z8spg\") pod \"01b87ad5-beef-41b7-99be-693434e57bbc\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") "
Nov 03 19:58:50.706773 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.706541    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-workload-certs\") pod \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") "
Nov 03 19:58:50.706773 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.706575    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-podinfo\") pod \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") "
Nov 03 19:58:50.706773 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.706664    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-istio-envoy\") pod \"01b87ad5-beef-41b7-99be-693434e57bbc\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") "
Nov 03 19:58:50.706773 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.706651    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-data" pod "2e0b0d0d-df65-4dfe-9333-1237cb41b8a4" (UID: "2e0b0d0d-df65-4dfe-9333-1237cb41b8a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:50.706773 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.706747    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01b87ad5-beef-41b7-99be-693434e57bbc-istio-token\" (UniqueName: \"kubernetes.io/projected/01b87ad5-beef-41b7-99be-693434e57bbc-istio-token\") pod \"01b87ad5-beef-41b7-99be-693434e57bbc\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") "
Nov 03 19:58:50.706773 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.706777    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-token\" (UniqueName: \"kubernetes.io/projected/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-token\") pod \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") "
Nov 03 19:58:50.707413 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.706812    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istiod-ca-cert" pod "2e0b0d0d-df65-4dfe-9333-1237cb41b8a4" (UID: "2e0b0d0d-df65-4dfe-9333-1237cb41b8a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:58:50.707413 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.706872    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01b87ad5-beef-41b7-99be-693434e57bbc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b87ad5-beef-41b7-99be-693434e57bbc-istiod-ca-cert\") pod \"01b87ad5-beef-41b7-99be-693434e57bbc\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") "
Nov 03 19:58:50.707413 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.706904    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-workload-certs" pod "2e0b0d0d-df65-4dfe-9333-1237cb41b8a4" (UID: "2e0b0d0d-df65-4dfe-9333-1237cb41b8a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:50.707413 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.706945    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-workload-socket\") pod \"01b87ad5-beef-41b7-99be-693434e57bbc\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") "
Nov 03 19:58:50.707413 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.707019    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-credential-socket\") pod \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\" (UID: \"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4\") "
Nov 03 19:58:50.707413 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.707048    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-istio-data\") pod \"01b87ad5-beef-41b7-99be-693434e57bbc\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") "
Nov 03 19:58:50.707413 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.707086    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-credential-socket\") pod \"01b87ad5-beef-41b7-99be-693434e57bbc\" (UID: \"01b87ad5-beef-41b7-99be-693434e57bbc\") "
Nov 03 19:58:50.707777 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.707584    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-workload-socket" pod "2e0b0d0d-df65-4dfe-9333-1237cb41b8a4" (UID: "2e0b0d0d-df65-4dfe-9333-1237cb41b8a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:50.707777 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.707631    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-workload-socket" pod "01b87ad5-beef-41b7-99be-693434e57bbc" (UID: "01b87ad5-beef-41b7-99be-693434e57bbc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:50.707899 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.707809    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-credential-socket" pod "2e0b0d0d-df65-4dfe-9333-1237cb41b8a4" (UID: "2e0b0d0d-df65-4dfe-9333-1237cb41b8a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:50.707957 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.707911    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-credential-socket" pod "01b87ad5-beef-41b7-99be-693434e57bbc" (UID: "01b87ad5-beef-41b7-99be-693434e57bbc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:50.707957 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.707919    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b87ad5-beef-41b7-99be-693434e57bbc-istiod-ca-cert" pod "01b87ad5-beef-41b7-99be-693434e57bbc" (UID: "01b87ad5-beef-41b7-99be-693434e57bbc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:58:50.707957 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.707921    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-istio-data" pod "01b87ad5-beef-41b7-99be-693434e57bbc" (UID: "01b87ad5-beef-41b7-99be-693434e57bbc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:50.708549 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.708524    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:50.708701 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.708554    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:50.708701 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.708572    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:50.708701 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.708587    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b87ad5-beef-41b7-99be-693434e57bbc-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:50.708701 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.708602    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:50.708701 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.708617    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:50.708701 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.708631    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:50.708701 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.708644    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:50.708701 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.708658    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:50.708701 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.708672    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:50.710179 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.709608    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-kube-api-access-8wkjl" pod "2e0b0d0d-df65-4dfe-9333-1237cb41b8a4" (UID: "2e0b0d0d-df65-4dfe-9333-1237cb41b8a4"). InnerVolumeSpecName "kube-api-access-8wkjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:58:50.710179 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.709698    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-istio-envoy" pod "01b87ad5-beef-41b7-99be-693434e57bbc" (UID: "01b87ad5-beef-41b7-99be-693434e57bbc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:50.710179 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.710132    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b87ad5-beef-41b7-99be-693434e57bbc-istio-token" pod "01b87ad5-beef-41b7-99be-693434e57bbc" (UID: "01b87ad5-beef-41b7-99be-693434e57bbc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:58:50.710179 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.710168    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-token" pod "2e0b0d0d-df65-4dfe-9333-1237cb41b8a4" (UID: "2e0b0d0d-df65-4dfe-9333-1237cb41b8a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:58:50.710459 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.710294    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-podinfo" pod "2e0b0d0d-df65-4dfe-9333-1237cb41b8a4" (UID: "2e0b0d0d-df65-4dfe-9333-1237cb41b8a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:58:50.710534 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.710510    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01b87ad5-beef-41b7-99be-693434e57bbc-istio-podinfo" pod "01b87ad5-beef-41b7-99be-693434e57bbc" (UID: "01b87ad5-beef-41b7-99be-693434e57bbc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:58:50.710807 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.710785    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-envoy" pod "2e0b0d0d-df65-4dfe-9333-1237cb41b8a4" (UID: "2e0b0d0d-df65-4dfe-9333-1237cb41b8a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:58:50.711351 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.711330    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b87ad5-beef-41b7-99be-693434e57bbc-kube-api-access-z8spg" pod "01b87ad5-beef-41b7-99be-693434e57bbc" (UID: "01b87ad5-beef-41b7-99be-693434e57bbc"). InnerVolumeSpecName "kube-api-access-z8spg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:58:50.810466 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.810381    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wkjl\" (UniqueName: \"kubernetes.io/projected/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-kube-api-access-8wkjl\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:50.810466 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.810412    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:50.810466 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.810424    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b87ad5-beef-41b7-99be-693434e57bbc-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:50.810466 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.810439    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8spg\" (UniqueName: \"kubernetes.io/projected/01b87ad5-beef-41b7-99be-693434e57bbc-kube-api-access-z8spg\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:50.810466 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.810447    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:50.810466 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.810456    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b87ad5-beef-41b7-99be-693434e57bbc-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:50.810466 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.810464    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01b87ad5-beef-41b7-99be-693434e57bbc-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:50.810466 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:50.810472    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:58:51.181922 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:51.181796    5591 generic.go:356] "Generic (PLEG): container finished" podID="01b87ad5-beef-41b7-99be-693434e57bbc" containerID="7fa297fc2046c52f70dd6a0e4b6cc424e082d815df3ee0d96a1a40cd628d33bf" exitCode=0
Nov 03 19:58:51.181922 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:51.181850    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"
Nov 03 19:58:51.181922 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:51.181899    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx" event={"ID":"01b87ad5-beef-41b7-99be-693434e57bbc","Type":"ContainerDied","Data":"7fa297fc2046c52f70dd6a0e4b6cc424e082d815df3ee0d96a1a40cd628d33bf"}
Nov 03 19:58:51.182326 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:51.181954    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx" event={"ID":"01b87ad5-beef-41b7-99be-693434e57bbc","Type":"ContainerDied","Data":"70d7d40b78539ff429a361df47c78965f0a5ece0cda13068b1f7a86063b61749"}
Nov 03 19:58:51.182326 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:51.181990    5591 scope.go:122] "RemoveContainer" containerID="7fa297fc2046c52f70dd6a0e4b6cc424e082d815df3ee0d96a1a40cd628d33bf"
Nov 03 19:58:51.183849 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:51.183824    5591 generic.go:356] "Generic (PLEG): container finished" podID="2e0b0d0d-df65-4dfe-9333-1237cb41b8a4" containerID="f451029784550ce6a6afc7134192e318ea92e5df08775922488f18a44b4363d0" exitCode=0
Nov 03 19:58:51.183972 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:51.183912    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7" event={"ID":"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4","Type":"ContainerDied","Data":"f451029784550ce6a6afc7134192e318ea92e5df08775922488f18a44b4363d0"}
Nov 03 19:58:51.184025 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:51.183982    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7" event={"ID":"2e0b0d0d-df65-4dfe-9333-1237cb41b8a4","Type":"ContainerDied","Data":"e5b32a89e1a79e3794736fc4dbe5de7540209bae4bfdc000fe47765f86820c87"}
Nov 03 19:58:51.184252 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:51.184194    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"
Nov 03 19:58:51.196870 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:51.196844    5591 scope.go:122] "RemoveContainer" containerID="7fa297fc2046c52f70dd6a0e4b6cc424e082d815df3ee0d96a1a40cd628d33bf"
Nov 03 19:58:51.197194 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:58:51.197176    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fa297fc2046c52f70dd6a0e4b6cc424e082d815df3ee0d96a1a40cd628d33bf\": container with ID starting with 7fa297fc2046c52f70dd6a0e4b6cc424e082d815df3ee0d96a1a40cd628d33bf not found: ID does not exist" containerID="7fa297fc2046c52f70dd6a0e4b6cc424e082d815df3ee0d96a1a40cd628d33bf"
Nov 03 19:58:51.197300 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:51.197221    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fa297fc2046c52f70dd6a0e4b6cc424e082d815df3ee0d96a1a40cd628d33bf"} err="failed to get container status \"7fa297fc2046c52f70dd6a0e4b6cc424e082d815df3ee0d96a1a40cd628d33bf\": rpc error: code = NotFound desc = could not find container \"7fa297fc2046c52f70dd6a0e4b6cc424e082d815df3ee0d96a1a40cd628d33bf\": container with ID starting with 7fa297fc2046c52f70dd6a0e4b6cc424e082d815df3ee0d96a1a40cd628d33bf not found: ID does not exist"
Nov 03 19:58:51.197300 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:51.197242    5591 scope.go:122] "RemoveContainer" containerID="f451029784550ce6a6afc7134192e318ea92e5df08775922488f18a44b4363d0"
Nov 03 19:58:51.208583 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:51.208564    5591 scope.go:122] "RemoveContainer" containerID="f451029784550ce6a6afc7134192e318ea92e5df08775922488f18a44b4363d0"
Nov 03 19:58:51.208916 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:58:51.208895    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f451029784550ce6a6afc7134192e318ea92e5df08775922488f18a44b4363d0\": container with ID starting with f451029784550ce6a6afc7134192e318ea92e5df08775922488f18a44b4363d0 not found: ID does not exist" containerID="f451029784550ce6a6afc7134192e318ea92e5df08775922488f18a44b4363d0"
Nov 03 19:58:51.208990 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:51.208924    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f451029784550ce6a6afc7134192e318ea92e5df08775922488f18a44b4363d0"} err="failed to get container status \"f451029784550ce6a6afc7134192e318ea92e5df08775922488f18a44b4363d0\": rpc error: code = NotFound desc = could not find container \"f451029784550ce6a6afc7134192e318ea92e5df08775922488f18a44b4363d0\": container with ID starting with f451029784550ce6a6afc7134192e318ea92e5df08775922488f18a44b4363d0 not found: ID does not exist"
Nov 03 19:58:51.247435 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:51.247405    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"]
Nov 03 19:58:51.274519 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:51.274486    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-92lb7"]
Nov 03 19:58:51.295166 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:51.295132    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"]
Nov 03 19:58:51.302291 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:51.302264    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-cvmbx"]
Nov 03 19:58:51.352363 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:51.352331    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01b87ad5-beef-41b7-99be-693434e57bbc" path="/var/lib/kubelet/pods/01b87ad5-beef-41b7-99be-693434e57bbc/volumes"
Nov 03 19:58:51.352857 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:51.352844    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e0b0d0d-df65-4dfe-9333-1237cb41b8a4" path="/var/lib/kubelet/pods/2e0b0d0d-df65-4dfe-9333-1237cb41b8a4/volumes"
Nov 03 19:58:51.878104 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:51.878076    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c"
Nov 03 19:58:51.878579 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:51.878149    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq"
Nov 03 19:58:52.371074 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.371043    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq"]
Nov 03 19:58:52.372989 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.372666    5591 cpu_manager.go:414] "containerMap: removing container" podUID="01b87ad5-beef-41b7-99be-693434e57bbc" containerName="istio-proxy"
Nov 03 19:58:52.372989 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.372909    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01b87ad5-beef-41b7-99be-693434e57bbc" containerName="istio-proxy"
Nov 03 19:58:52.372989 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.372977    5591 cpu_manager.go:414] "containerMap: removing container" podUID="2e0b0d0d-df65-4dfe-9333-1237cb41b8a4" containerName="istio-proxy"
Nov 03 19:58:52.372989 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.372988    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e0b0d0d-df65-4dfe-9333-1237cb41b8a4" containerName="istio-proxy"
Nov 03 19:58:52.373372 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.373191    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e0b0d0d-df65-4dfe-9333-1237cb41b8a4" containerName="istio-proxy"
Nov 03 19:58:52.373372 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.373242    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="01b87ad5-beef-41b7-99be-693434e57bbc" containerName="istio-proxy"
Nov 03 19:58:52.377923 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.377901    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq"
Nov 03 19:58:52.392047 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.392020    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-f493f7-serving-cert\""
Nov 03 19:58:52.396057 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.396031    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq"]
Nov 03 19:58:52.541853 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.541767    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzgg8\" (UniqueName: \"kubernetes.io/projected/fdab0164-a331-4abe-a4ed-f5c299bde686-kube-api-access-pzgg8\") pod \"maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq\" (UID: \"fdab0164-a331-4abe-a4ed-f5c299bde686\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq"
Nov 03 19:58:52.541853 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.541827    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdab0164-a331-4abe-a4ed-f5c299bde686-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq\" (UID: \"fdab0164-a331-4abe-a4ed-f5c299bde686\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq"
Nov 03 19:58:52.542046 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.541996    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdab0164-a331-4abe-a4ed-f5c299bde686-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq\" (UID: \"fdab0164-a331-4abe-a4ed-f5c299bde686\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq"
Nov 03 19:58:52.542257 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.542192    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdab0164-a331-4abe-a4ed-f5c299bde686-metrics-certs\") pod \"maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq\" (UID: \"fdab0164-a331-4abe-a4ed-f5c299bde686\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq"
Nov 03 19:58:52.651334 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.644722    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzgg8\" (UniqueName: \"kubernetes.io/projected/fdab0164-a331-4abe-a4ed-f5c299bde686-kube-api-access-pzgg8\") pod \"maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq\" (UID: \"fdab0164-a331-4abe-a4ed-f5c299bde686\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq"
Nov 03 19:58:52.651334 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.644817    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdab0164-a331-4abe-a4ed-f5c299bde686-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq\" (UID: \"fdab0164-a331-4abe-a4ed-f5c299bde686\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq"
Nov 03 19:58:52.651334 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.644981    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdab0164-a331-4abe-a4ed-f5c299bde686-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq\" (UID: \"fdab0164-a331-4abe-a4ed-f5c299bde686\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq"
Nov 03 19:58:52.651334 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.645195    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdab0164-a331-4abe-a4ed-f5c299bde686-metrics-certs\") pod \"maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq\" (UID: \"fdab0164-a331-4abe-a4ed-f5c299bde686\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq"
Nov 03 19:58:52.698709 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.697849    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdab0164-a331-4abe-a4ed-f5c299bde686-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq\" (UID: \"fdab0164-a331-4abe-a4ed-f5c299bde686\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq"
Nov 03 19:58:52.712252 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.712220    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdab0164-a331-4abe-a4ed-f5c299bde686-metrics-certs\") pod \"maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq\" (UID: \"fdab0164-a331-4abe-a4ed-f5c299bde686\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq"
Nov 03 19:58:52.753281 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.750117    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdab0164-a331-4abe-a4ed-f5c299bde686-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq\" (UID: \"fdab0164-a331-4abe-a4ed-f5c299bde686\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq"
Nov 03 19:58:52.801515 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.801354    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzgg8\" (UniqueName: \"kubernetes.io/projected/fdab0164-a331-4abe-a4ed-f5c299bde686-kube-api-access-pzgg8\") pod \"maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq\" (UID: \"fdab0164-a331-4abe-a4ed-f5c299bde686\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq"
Nov 03 19:58:52.846442 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.846404    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m"]
Nov 03 19:58:52.850044 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.850024    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m"
Nov 03 19:58:52.869948 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.869919    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m"]
Nov 03 19:58:52.892384 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.892346    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-3faa07\""
Nov 03 19:58:52.901888 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.901859    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68"]
Nov 03 19:58:52.905295 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.905260    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68"
Nov 03 19:58:52.943722 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.943687    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-3faa07-dockercfg-k6cps\""
Nov 03 19:58:52.950100 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.950070    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68"]
Nov 03 19:58:52.992061 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.992014    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhtqw\" (UniqueName: \"kubernetes.io/projected/270abfe6-e1dc-4a2e-a106-4cd071d79612-kube-api-access-dhtqw\") pod \"payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m\" (UID: \"270abfe6-e1dc-4a2e-a106-4cd071d79612\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m"
Nov 03 19:58:52.992282 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:52.992244    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/270abfe6-e1dc-4a2e-a106-4cd071d79612-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m\" (UID: \"270abfe6-e1dc-4a2e-a106-4cd071d79612\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m"
Nov 03 19:58:53.037804 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:53.018514    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq"
Nov 03 19:58:53.138245 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:53.135586    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhtqw\" (UniqueName: \"kubernetes.io/projected/270abfe6-e1dc-4a2e-a106-4cd071d79612-kube-api-access-dhtqw\") pod \"payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m\" (UID: \"270abfe6-e1dc-4a2e-a106-4cd071d79612\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m"
Nov 03 19:58:53.138245 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:53.135852    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/270abfe6-e1dc-4a2e-a106-4cd071d79612-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m\" (UID: \"270abfe6-e1dc-4a2e-a106-4cd071d79612\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m"
Nov 03 19:58:53.138245 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:53.135891    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74cbn\" (UniqueName: \"kubernetes.io/projected/4dc9dccb-784a-4082-9a7f-39223a74ce3b-kube-api-access-74cbn\") pod \"payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68\" (UID: \"4dc9dccb-784a-4082-9a7f-39223a74ce3b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68"
Nov 03 19:58:53.138245 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:53.135951    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc9dccb-784a-4082-9a7f-39223a74ce3b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68\" (UID: \"4dc9dccb-784a-4082-9a7f-39223a74ce3b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68"
Nov 03 19:58:53.172245 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:53.170391    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhtqw\" (UniqueName: \"kubernetes.io/projected/270abfe6-e1dc-4a2e-a106-4cd071d79612-kube-api-access-dhtqw\") pod \"payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m\" (UID: \"270abfe6-e1dc-4a2e-a106-4cd071d79612\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m"
Nov 03 19:58:53.184252 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:53.178596    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/270abfe6-e1dc-4a2e-a106-4cd071d79612-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m\" (UID: \"270abfe6-e1dc-4a2e-a106-4cd071d79612\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m"
Nov 03 19:58:53.242272 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:53.239472    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74cbn\" (UniqueName: \"kubernetes.io/projected/4dc9dccb-784a-4082-9a7f-39223a74ce3b-kube-api-access-74cbn\") pod \"payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68\" (UID: \"4dc9dccb-784a-4082-9a7f-39223a74ce3b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68"
Nov 03 19:58:53.242272 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:53.239549    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc9dccb-784a-4082-9a7f-39223a74ce3b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68\" (UID: \"4dc9dccb-784a-4082-9a7f-39223a74ce3b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68"
Nov 03 19:58:53.242272 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:53.240670    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc9dccb-784a-4082-9a7f-39223a74ce3b-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68\" (UID: \"4dc9dccb-784a-4082-9a7f-39223a74ce3b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68"
Nov 03 19:58:53.293243 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:53.291035    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74cbn\" (UniqueName: \"kubernetes.io/projected/4dc9dccb-784a-4082-9a7f-39223a74ce3b-kube-api-access-74cbn\") pod \"payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68\" (UID: \"4dc9dccb-784a-4082-9a7f-39223a74ce3b\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68"
Nov 03 19:58:53.469846 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:53.469787    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m"
Nov 03 19:58:53.558325 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:53.554375    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68"
Nov 03 19:58:53.779919 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:53.775835    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq"]
Nov 03 19:58:53.940549 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:53.940376    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh"
Nov 03 19:58:53.940549 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:53.940545    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4"
Nov 03 19:58:54.048275 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:54.046600    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68"]
Nov 03 19:58:54.104570 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:54.104545    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m"]
Nov 03 19:58:54.107836 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:58:54.107803    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod270abfe6_e1dc_4a2e_a106_4cd071d79612.slice/crio-c582a1ff8f1d8f9c731e3e7ed4f1b03fa6a1b7a0f0c49af17b01e7bcc26a1f17 WatchSource:0}: Error finding container c582a1ff8f1d8f9c731e3e7ed4f1b03fa6a1b7a0f0c49af17b01e7bcc26a1f17: Status 404 returned error can't find the container with id c582a1ff8f1d8f9c731e3e7ed4f1b03fa6a1b7a0f0c49af17b01e7bcc26a1f17
Nov 03 19:58:54.241095 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:54.240995    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m" event={"ID":"270abfe6-e1dc-4a2e-a106-4cd071d79612","Type":"ContainerStarted","Data":"d8d4f00d80fb6ad4ac1f035cb304f827106df005ad0770a87a27b6d279b16fdc"}
Nov 03 19:58:54.241095 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:54.241049    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m" event={"ID":"270abfe6-e1dc-4a2e-a106-4cd071d79612","Type":"ContainerStarted","Data":"c582a1ff8f1d8f9c731e3e7ed4f1b03fa6a1b7a0f0c49af17b01e7bcc26a1f17"}
Nov 03 19:58:54.241369 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:54.241169    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m"
Nov 03 19:58:54.242511 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:54.242486    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq" event={"ID":"fdab0164-a331-4abe-a4ed-f5c299bde686","Type":"ContainerStarted","Data":"7362b3afca0aec3643c24e4264cba9e30a129b1270d797f0101ac55ea0aa3fa0"}
Nov 03 19:58:54.244056 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:54.244032    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68" event={"ID":"4dc9dccb-784a-4082-9a7f-39223a74ce3b","Type":"ContainerStarted","Data":"935f93555316f512cb6ccfa004ba4572b8a453e30499e40c1d563f3cb977f2f0"}
Nov 03 19:58:54.244175 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:54.244069    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68" event={"ID":"4dc9dccb-784a-4082-9a7f-39223a74ce3b","Type":"ContainerStarted","Data":"fff137e6ee0aef27271b0d80471d8b4b08c6d458ddbb7986c57caa90c8cda4af"}
Nov 03 19:58:54.244175 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:54.244101    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68"
Nov 03 19:58:54.342232 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:54.338753    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m" podStartSLOduration=2.338721479 podStartE2EDuration="2.338721479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:54.314991598 +0000 UTC m=+1283.087284617" watchObservedRunningTime="2026-11-03 19:58:54.338721479 +0000 UTC m=+1283.111014497"
Nov 03 19:58:54.387245 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:54.384805    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68" podStartSLOduration=2.38478922 podStartE2EDuration="2.38478922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:54.370243837 +0000 UTC m=+1283.142536856" watchObservedRunningTime="2026-11-03 19:58:54.38478922 +0000 UTC m=+1283.157082239"
Nov 03 19:58:55.255894 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:55.255848    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq" event={"ID":"fdab0164-a331-4abe-a4ed-f5c299bde686","Type":"ContainerStarted","Data":"45d155c9f013036b320c35c58b5f1b4724771e46fa5b5f79cc708c18747b90d9"}
Nov 03 19:58:55.256380 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:55.256093    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq"
Nov 03 19:58:55.306606 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:55.306550    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq" podStartSLOduration=2.839220478 podStartE2EDuration="3.306534169s" totalImagesPullingTime="467.313691ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:55.299748357 +0000 UTC m=+1284.072041377" watchObservedRunningTime="2026-11-03 19:58:55.306534169 +0000 UTC m=+1284.078827187"
Nov 03 19:58:55.471099 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:55.470986    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r"]
Nov 03 19:58:55.536230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:55.531089    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r"]
Nov 03 19:58:55.536230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:55.531260    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r"
Nov 03 19:58:55.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:55.558662    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-277dc9-serving-cert\""
Nov 03 19:58:55.579272 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:55.579195    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48822eec-0744-4d8b-983a-9129d93987fc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r\" (UID: \"48822eec-0744-4d8b-983a-9129d93987fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r"
Nov 03 19:58:55.579502 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:55.579459    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48822eec-0744-4d8b-983a-9129d93987fc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r\" (UID: \"48822eec-0744-4d8b-983a-9129d93987fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r"
Nov 03 19:58:55.579640 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:55.579607    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6889d\" (UniqueName: \"kubernetes.io/projected/48822eec-0744-4d8b-983a-9129d93987fc-kube-api-access-6889d\") pod \"maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r\" (UID: \"48822eec-0744-4d8b-983a-9129d93987fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r"
Nov 03 19:58:55.579758 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:55.579742    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48822eec-0744-4d8b-983a-9129d93987fc-metrics-certs\") pod \"maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r\" (UID: \"48822eec-0744-4d8b-983a-9129d93987fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r"
Nov 03 19:58:55.687262 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:55.686311    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48822eec-0744-4d8b-983a-9129d93987fc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r\" (UID: \"48822eec-0744-4d8b-983a-9129d93987fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r"
Nov 03 19:58:55.687262 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:55.686446    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6889d\" (UniqueName: \"kubernetes.io/projected/48822eec-0744-4d8b-983a-9129d93987fc-kube-api-access-6889d\") pod \"maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r\" (UID: \"48822eec-0744-4d8b-983a-9129d93987fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r"
Nov 03 19:58:55.687262 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:55.686706    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48822eec-0744-4d8b-983a-9129d93987fc-metrics-certs\") pod \"maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r\" (UID: \"48822eec-0744-4d8b-983a-9129d93987fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r"
Nov 03 19:58:55.687262 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:55.686833    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48822eec-0744-4d8b-983a-9129d93987fc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r\" (UID: \"48822eec-0744-4d8b-983a-9129d93987fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r"
Nov 03 19:58:55.692227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:55.690361    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48822eec-0744-4d8b-983a-9129d93987fc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r\" (UID: \"48822eec-0744-4d8b-983a-9129d93987fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r"
Nov 03 19:58:55.692227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:55.691561    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48822eec-0744-4d8b-983a-9129d93987fc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r\" (UID: \"48822eec-0744-4d8b-983a-9129d93987fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r"
Nov 03 19:58:55.724227 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:55.718851    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48822eec-0744-4d8b-983a-9129d93987fc-metrics-certs\") pod \"maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r\" (UID: \"48822eec-0744-4d8b-983a-9129d93987fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r"
Nov 03 19:58:55.745226 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:55.726643    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6889d\" (UniqueName: \"kubernetes.io/projected/48822eec-0744-4d8b-983a-9129d93987fc-kube-api-access-6889d\") pod \"maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r\" (UID: \"48822eec-0744-4d8b-983a-9129d93987fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r"
Nov 03 19:58:55.891415 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:55.891329    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r"
Nov 03 19:58:56.234245 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:56.234215    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r"]
Nov 03 19:58:56.235799 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:58:56.235773    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48822eec_0744_4d8b_983a_9129d93987fc.slice/crio-4a8bc811817f99068bb8c09f1997e199c6dcd9e3ccb21f2e6c2eb22c4683fc72 WatchSource:0}: Error finding container 4a8bc811817f99068bb8c09f1997e199c6dcd9e3ccb21f2e6c2eb22c4683fc72: Status 404 returned error can't find the container with id 4a8bc811817f99068bb8c09f1997e199c6dcd9e3ccb21f2e6c2eb22c4683fc72
Nov 03 19:58:56.383236 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:56.376930    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r" event={"ID":"48822eec-0744-4d8b-983a-9129d93987fc","Type":"ContainerStarted","Data":"4a8bc811817f99068bb8c09f1997e199c6dcd9e3ccb21f2e6c2eb22c4683fc72"}
Nov 03 19:58:57.386398 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:57.386359    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r" event={"ID":"48822eec-0744-4d8b-983a-9129d93987fc","Type":"ContainerStarted","Data":"190c3df423a3809f4a58b98bfcc54d2284c7ac3b11199d403c6b7eb06bd55f12"}
Nov 03 19:58:57.386818 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:57.386436    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r"
Nov 03 19:58:57.407703 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:58:57.407640    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r" podStartSLOduration=1.913950101 podStartE2EDuration="2.407622497s" totalImagesPullingTime="493.672396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:58:57.406954948 +0000 UTC m=+1286.179247963" watchObservedRunningTime="2026-11-03 19:58:57.407622497 +0000 UTC m=+1286.179915517"
Nov 03 19:58:58.949244 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:58:58.946856    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:59:01.389183 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:01.389153    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq"
Nov 03 19:59:03.398696 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:03.398666    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r"
Nov 03 19:59:05.150637 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:05.150598    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"]
Nov 03 19:59:05.151056 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:05.150895    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx" podUID="41a51c01-3bd6-4414-9f7e-6347d2d59727" containerName="istio-proxy" containerID="cri-o://4d63baa9ecee0765a1c52a31926b742572329be1668603073303ae7aaf64ecec" gracePeriod=30
Nov 03 19:59:05.333231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:05.326459    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68"
Nov 03 19:59:05.343228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:05.334402    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m"
Nov 03 19:59:10.602249 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.601679    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:59:10.728340 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.697731    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-token\" (UniqueName: \"kubernetes.io/projected/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-token\") pod \"41a51c01-3bd6-4414-9f7e-6347d2d59727\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") "
Nov 03 19:59:10.728340 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.697843    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41a51c01-3bd6-4414-9f7e-6347d2d59727-kube-api-access-z2hjs\" (UniqueName: \"kubernetes.io/projected/41a51c01-3bd6-4414-9f7e-6347d2d59727-kube-api-access-z2hjs\") pod \"41a51c01-3bd6-4414-9f7e-6347d2d59727\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") "
Nov 03 19:59:10.728340 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.697902    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-credential-socket\") pod \"41a51c01-3bd6-4414-9f7e-6347d2d59727\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") "
Nov 03 19:59:10.728340 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.697944    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-data\") pod \"41a51c01-3bd6-4414-9f7e-6347d2d59727\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") "
Nov 03 19:59:10.728340 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.698002    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-podinfo\") pod \"41a51c01-3bd6-4414-9f7e-6347d2d59727\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") "
Nov 03 19:59:10.728340 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.698048    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-envoy\") pod \"41a51c01-3bd6-4414-9f7e-6347d2d59727\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") "
Nov 03 19:59:10.728340 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.698231    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-workload-socket\") pod \"41a51c01-3bd6-4414-9f7e-6347d2d59727\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") "
Nov 03 19:59:10.728340 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.698364    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41a51c01-3bd6-4414-9f7e-6347d2d59727-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a51c01-3bd6-4414-9f7e-6347d2d59727-istiod-ca-cert\") pod \"41a51c01-3bd6-4414-9f7e-6347d2d59727\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") "
Nov 03 19:59:10.728340 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.698438    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-workload-certs\") pod \"41a51c01-3bd6-4414-9f7e-6347d2d59727\" (UID: \"41a51c01-3bd6-4414-9f7e-6347d2d59727\") "
Nov 03 19:59:10.753230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.729239    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-workload-certs" pod "41a51c01-3bd6-4414-9f7e-6347d2d59727" (UID: "41a51c01-3bd6-4414-9f7e-6347d2d59727"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:59:10.786228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.763442    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41a51c01-3bd6-4414-9f7e-6347d2d59727-istiod-ca-cert" pod "41a51c01-3bd6-4414-9f7e-6347d2d59727" (UID: "41a51c01-3bd6-4414-9f7e-6347d2d59727"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:59:10.786228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.784418    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-token" pod "41a51c01-3bd6-4414-9f7e-6347d2d59727" (UID: "41a51c01-3bd6-4414-9f7e-6347d2d59727"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:59:10.814242 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.786657    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-workload-socket" pod "41a51c01-3bd6-4414-9f7e-6347d2d59727" (UID: "41a51c01-3bd6-4414-9f7e-6347d2d59727"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:59:10.814242 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.789299    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-credential-socket" pod "41a51c01-3bd6-4414-9f7e-6347d2d59727" (UID: "41a51c01-3bd6-4414-9f7e-6347d2d59727"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:59:10.830537 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.818363    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-envoy" pod "41a51c01-3bd6-4414-9f7e-6347d2d59727" (UID: "41a51c01-3bd6-4414-9f7e-6347d2d59727"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:59:10.879277 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.873281    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a51c01-3bd6-4414-9f7e-6347d2d59727-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:10.879277 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.873316    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:10.879277 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.873329    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:10.879277 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.873341    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:10.879277 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.873354    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:10.879277 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.875311    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-data" pod "41a51c01-3bd6-4414-9f7e-6347d2d59727" (UID: "41a51c01-3bd6-4414-9f7e-6347d2d59727"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:59:10.915838 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.915442    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-podinfo" pod "41a51c01-3bd6-4414-9f7e-6347d2d59727" (UID: "41a51c01-3bd6-4414-9f7e-6347d2d59727"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:59:10.973006 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.972975    5591 generic.go:356] "Generic (PLEG): container finished" podID="41a51c01-3bd6-4414-9f7e-6347d2d59727" containerID="4d63baa9ecee0765a1c52a31926b742572329be1668603073303ae7aaf64ecec" exitCode=0
Nov 03 19:59:10.973193 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.973031    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"
Nov 03 19:59:10.973193 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.973052    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx" event={"ID":"41a51c01-3bd6-4414-9f7e-6347d2d59727","Type":"ContainerDied","Data":"4d63baa9ecee0765a1c52a31926b742572329be1668603073303ae7aaf64ecec"}
Nov 03 19:59:10.973193 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.973106    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx" event={"ID":"41a51c01-3bd6-4414-9f7e-6347d2d59727","Type":"ContainerDied","Data":"7c060aac16400f4d3a8836f3de3163355cbedafa9161e6f4657b3a202042ba47"}
Nov 03 19:59:10.973193 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.973138    5591 scope.go:122] "RemoveContainer" containerID="4d63baa9ecee0765a1c52a31926b742572329be1668603073303ae7aaf64ecec"
Nov 03 19:59:10.975028 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.974523    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:10.975028 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.974541    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:10.975028 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.974551    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a51c01-3bd6-4414-9f7e-6347d2d59727-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:10.976152 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.976127    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a51c01-3bd6-4414-9f7e-6347d2d59727-kube-api-access-z2hjs" pod "41a51c01-3bd6-4414-9f7e-6347d2d59727" (UID: "41a51c01-3bd6-4414-9f7e-6347d2d59727"). InnerVolumeSpecName "kube-api-access-z2hjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:59:10.990313 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:10.990292    5591 scope.go:122] "RemoveContainer" containerID="207a93fb3a8b4488f693bbc0b80e459f5e849c83c45704d7787db623441feff0"
Nov 03 19:59:11.000910 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:11.000887    5591 scope.go:122] "RemoveContainer" containerID="4d63baa9ecee0765a1c52a31926b742572329be1668603073303ae7aaf64ecec"
Nov 03 19:59:11.001271 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:59:11.001249    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d63baa9ecee0765a1c52a31926b742572329be1668603073303ae7aaf64ecec\": container with ID starting with 4d63baa9ecee0765a1c52a31926b742572329be1668603073303ae7aaf64ecec not found: ID does not exist" containerID="4d63baa9ecee0765a1c52a31926b742572329be1668603073303ae7aaf64ecec"
Nov 03 19:59:11.001323 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:11.001283    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d63baa9ecee0765a1c52a31926b742572329be1668603073303ae7aaf64ecec"} err="failed to get container status \"4d63baa9ecee0765a1c52a31926b742572329be1668603073303ae7aaf64ecec\": rpc error: code = NotFound desc = could not find container \"4d63baa9ecee0765a1c52a31926b742572329be1668603073303ae7aaf64ecec\": container with ID starting with 4d63baa9ecee0765a1c52a31926b742572329be1668603073303ae7aaf64ecec not found: ID does not exist"
Nov 03 19:59:11.001323 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:11.001302    5591 scope.go:122] "RemoveContainer" containerID="207a93fb3a8b4488f693bbc0b80e459f5e849c83c45704d7787db623441feff0"
Nov 03 19:59:11.001649 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:59:11.001616    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"207a93fb3a8b4488f693bbc0b80e459f5e849c83c45704d7787db623441feff0\": container with ID starting with 207a93fb3a8b4488f693bbc0b80e459f5e849c83c45704d7787db623441feff0 not found: ID does not exist" containerID="207a93fb3a8b4488f693bbc0b80e459f5e849c83c45704d7787db623441feff0"
Nov 03 19:59:11.001697 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:11.001656    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"207a93fb3a8b4488f693bbc0b80e459f5e849c83c45704d7787db623441feff0"} err="failed to get container status \"207a93fb3a8b4488f693bbc0b80e459f5e849c83c45704d7787db623441feff0\": rpc error: code = NotFound desc = could not find container \"207a93fb3a8b4488f693bbc0b80e459f5e849c83c45704d7787db623441feff0\": container with ID starting with 207a93fb3a8b4488f693bbc0b80e459f5e849c83c45704d7787db623441feff0 not found: ID does not exist"
Nov 03 19:59:11.076732 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:11.076700    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2hjs\" (UniqueName: \"kubernetes.io/projected/41a51c01-3bd6-4414-9f7e-6347d2d59727-kube-api-access-z2hjs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:11.301512 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:11.301470    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"]
Nov 03 19:59:11.304861 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:11.304835    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-29f633-openshift-default-68b8bcc957-7mmqx"]
Nov 03 19:59:11.351000 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:11.350965    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41a51c01-3bd6-4414-9f7e-6347d2d59727" path="/var/lib/kubelet/pods/41a51c01-3bd6-4414-9f7e-6347d2d59727/volumes"
Nov 03 19:59:11.859315 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:11.859278    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx"]
Nov 03 19:59:11.862109 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:11.860241    5591 cpu_manager.go:414] "containerMap: removing container" podUID="41a51c01-3bd6-4414-9f7e-6347d2d59727" containerName="istio-proxy"
Nov 03 19:59:11.862109 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:11.860262    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41a51c01-3bd6-4414-9f7e-6347d2d59727" containerName="istio-proxy"
Nov 03 19:59:11.862109 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:11.860310    5591 cpu_manager.go:414] "containerMap: removing container" podUID="41a51c01-3bd6-4414-9f7e-6347d2d59727" containerName="istio-proxy"
Nov 03 19:59:11.862109 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:11.860317    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41a51c01-3bd6-4414-9f7e-6347d2d59727" containerName="istio-proxy"
Nov 03 19:59:11.862109 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:11.860477    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="41a51c01-3bd6-4414-9f7e-6347d2d59727" containerName="istio-proxy"
Nov 03 19:59:11.862109 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:11.860494    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="41a51c01-3bd6-4414-9f7e-6347d2d59727" containerName="istio-proxy"
Nov 03 19:59:11.866549 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:11.866526    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx"
Nov 03 19:59:11.873519 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:11.873477    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx"]
Nov 03 19:59:11.875010 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:11.874983    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c6092e-dockercfg-rgld5\""
Nov 03 19:59:11.886158 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:11.886127    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw"]
Nov 03 19:59:11.891590 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:11.891550    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw"
Nov 03 19:59:11.903322 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:11.902358    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw"]
Nov 03 19:59:11.930898 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:11.930867    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c6092e\""
Nov 03 19:59:12.024250 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:12.018479    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbkv8\" (UniqueName: \"kubernetes.io/projected/8656a4cc-537b-47c7-9429-6581fb9ab565-kube-api-access-pbkv8\") pod \"payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw\" (UID: \"8656a4cc-537b-47c7-9429-6581fb9ab565\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw"
Nov 03 19:59:12.024250 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:12.018610    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsghk\" (UniqueName: \"kubernetes.io/projected/5522a57b-9892-4cd8-849e-1c68b47ea721-kube-api-access-hsghk\") pod \"payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx\" (UID: \"5522a57b-9892-4cd8-849e-1c68b47ea721\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx"
Nov 03 19:59:12.024250 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:12.018657    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5522a57b-9892-4cd8-849e-1c68b47ea721-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx\" (UID: \"5522a57b-9892-4cd8-849e-1c68b47ea721\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx"
Nov 03 19:59:12.024250 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:12.018941    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8656a4cc-537b-47c7-9429-6581fb9ab565-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw\" (UID: \"8656a4cc-537b-47c7-9429-6581fb9ab565\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw"
Nov 03 19:59:12.139297 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:12.122467    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8656a4cc-537b-47c7-9429-6581fb9ab565-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw\" (UID: \"8656a4cc-537b-47c7-9429-6581fb9ab565\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw"
Nov 03 19:59:12.139297 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:12.122609    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbkv8\" (UniqueName: \"kubernetes.io/projected/8656a4cc-537b-47c7-9429-6581fb9ab565-kube-api-access-pbkv8\") pod \"payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw\" (UID: \"8656a4cc-537b-47c7-9429-6581fb9ab565\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw"
Nov 03 19:59:12.139297 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:12.122703    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsghk\" (UniqueName: \"kubernetes.io/projected/5522a57b-9892-4cd8-849e-1c68b47ea721-kube-api-access-hsghk\") pod \"payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx\" (UID: \"5522a57b-9892-4cd8-849e-1c68b47ea721\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx"
Nov 03 19:59:12.139297 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:12.122752    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5522a57b-9892-4cd8-849e-1c68b47ea721-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx\" (UID: \"5522a57b-9892-4cd8-849e-1c68b47ea721\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx"
Nov 03 19:59:12.139297 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:12.123842    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5522a57b-9892-4cd8-849e-1c68b47ea721-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx\" (UID: \"5522a57b-9892-4cd8-849e-1c68b47ea721\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx"
Nov 03 19:59:12.156265 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:12.140393    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8656a4cc-537b-47c7-9429-6581fb9ab565-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw\" (UID: \"8656a4cc-537b-47c7-9429-6581fb9ab565\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw"
Nov 03 19:59:12.179231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:12.179178    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbkv8\" (UniqueName: \"kubernetes.io/projected/8656a4cc-537b-47c7-9429-6581fb9ab565-kube-api-access-pbkv8\") pod \"payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw\" (UID: \"8656a4cc-537b-47c7-9429-6581fb9ab565\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw"
Nov 03 19:59:12.203241 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:12.202945    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsghk\" (UniqueName: \"kubernetes.io/projected/5522a57b-9892-4cd8-849e-1c68b47ea721-kube-api-access-hsghk\") pod \"payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx\" (UID: \"5522a57b-9892-4cd8-849e-1c68b47ea721\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx"
Nov 03 19:59:12.241598 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:12.241257    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw"
Nov 03 19:59:12.486395 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:12.486295    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx"
Nov 03 19:59:12.602373 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:12.602340    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw"]
Nov 03 19:59:12.810268 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:12.810235    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx"]
Nov 03 19:59:13.134390 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:13.134296    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw" event={"ID":"8656a4cc-537b-47c7-9429-6581fb9ab565","Type":"ContainerStarted","Data":"181a37cbef8b4250153447fa2770ae4479375d0b2b7a21db97d31cce259f0048"}
Nov 03 19:59:13.135874 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:13.135843    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw" event={"ID":"8656a4cc-537b-47c7-9429-6581fb9ab565","Type":"ContainerStarted","Data":"6fd38ab523a0c52b3126e0e183b26a22d44a3f60be99875ea412b614fd71d35b"}
Nov 03 19:59:13.137356 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:13.137312    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw"
Nov 03 19:59:13.139729 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:13.139705    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx" event={"ID":"5522a57b-9892-4cd8-849e-1c68b47ea721","Type":"ContainerStarted","Data":"7cff7ce7400f0df20e3ce15481b48c1a624a58a22da264124f8f3691e0e34d5f"}
Nov 03 19:59:13.139916 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:13.139900    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx" event={"ID":"5522a57b-9892-4cd8-849e-1c68b47ea721","Type":"ContainerStarted","Data":"54b023f4518ad30d3dc850cb3f577072ba26202c6b0f91bba0c4fc5acd44dc7c"}
Nov 03 19:59:13.140956 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:13.140896    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx"
Nov 03 19:59:13.166372 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:13.166310    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw" podStartSLOduration=2.16628894 podStartE2EDuration="2.16628894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:13.164465885 +0000 UTC m=+1301.936758904" watchObservedRunningTime="2026-11-03 19:59:13.16628894 +0000 UTC m=+1301.938581959"
Nov 03 19:59:13.204601 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:13.204545    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx" podStartSLOduration=2.204529146 podStartE2EDuration="2.204529146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:13.197870729 +0000 UTC m=+1301.970163748" watchObservedRunningTime="2026-11-03 19:59:13.204529146 +0000 UTC m=+1301.976822163"
Nov 03 19:59:14.415857 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:14.415821    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh"]
Nov 03 19:59:14.429642 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:14.429603    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4"]
Nov 03 19:59:17.616749 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:17.615549    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"]
Nov 03 19:59:17.616749 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:17.615682    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"]
Nov 03 19:59:17.616749 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:17.615928    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc" podUID="770df5ab-84a7-4fe9-999b-8ca9ee6464a4" containerName="istio-proxy" containerID="cri-o://05986903284faf908de90f1f43aab54545c06a3e029af3a54a2d6c3731028307" gracePeriod=30
Nov 03 19:59:17.616749 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:17.616328    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds" podUID="3377787a-22d0-45ce-9c24-c39c840af373" containerName="istio-proxy" containerID="cri-o://d07698bd0f99375147174643e3f8fd9636b15d9f5f0e389a08272a36c44231eb" gracePeriod=30
Nov 03 19:59:19.416312 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:19.416261    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh" podUID="db5bfc00-f082-405e-8d43-843cbaf95b52" containerName="payload-processing" containerID="cri-o://62f7076414d6d48e66c1196cca0beddf60c3285d705c1e48dec5cf9e17b4c7c2" gracePeriod=25
Nov 03 19:59:19.430836 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:19.430785    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4" podUID="309d5f96-0577-45a6-8855-727412314a51" containerName="payload-pre-processing" containerID="cri-o://53bf3186af373d21a94c9eebf3a15b8964c59391ca81a814976d5f734b895c7a" gracePeriod=25
Nov 03 19:59:19.808758 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:19.808730    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh"
Nov 03 19:59:19.926062 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:19.926038    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4"
Nov 03 19:59:19.931184 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:19.931159    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db5bfc00-f082-405e-8d43-843cbaf95b52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db5bfc00-f082-405e-8d43-843cbaf95b52-plugins-config-volume\") pod \"db5bfc00-f082-405e-8d43-843cbaf95b52\" (UID: \"db5bfc00-f082-405e-8d43-843cbaf95b52\") "
Nov 03 19:59:19.931377 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:19.931361    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db5bfc00-f082-405e-8d43-843cbaf95b52-kube-api-access-8lvch\" (UniqueName: \"kubernetes.io/projected/db5bfc00-f082-405e-8d43-843cbaf95b52-kube-api-access-8lvch\") pod \"db5bfc00-f082-405e-8d43-843cbaf95b52\" (UID: \"db5bfc00-f082-405e-8d43-843cbaf95b52\") "
Nov 03 19:59:19.931599 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:19.931575    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db5bfc00-f082-405e-8d43-843cbaf95b52-plugins-config-volume" pod "db5bfc00-f082-405e-8d43-843cbaf95b52" (UID: "db5bfc00-f082-405e-8d43-843cbaf95b52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:59:19.931923 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:19.931902    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db5bfc00-f082-405e-8d43-843cbaf95b52-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:19.933671 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:19.933651    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db5bfc00-f082-405e-8d43-843cbaf95b52-kube-api-access-8lvch" pod "db5bfc00-f082-405e-8d43-843cbaf95b52" (UID: "db5bfc00-f082-405e-8d43-843cbaf95b52"). InnerVolumeSpecName "kube-api-access-8lvch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:59:20.032852 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:20.032815    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/309d5f96-0577-45a6-8855-727412314a51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/309d5f96-0577-45a6-8855-727412314a51-plugins-config-volume\") pod \"309d5f96-0577-45a6-8855-727412314a51\" (UID: \"309d5f96-0577-45a6-8855-727412314a51\") "
Nov 03 19:59:20.033049 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:20.032892    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/309d5f96-0577-45a6-8855-727412314a51-kube-api-access-tvt77\" (UniqueName: \"kubernetes.io/projected/309d5f96-0577-45a6-8855-727412314a51-kube-api-access-tvt77\") pod \"309d5f96-0577-45a6-8855-727412314a51\" (UID: \"309d5f96-0577-45a6-8855-727412314a51\") "
Nov 03 19:59:20.033353 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:20.033326    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/309d5f96-0577-45a6-8855-727412314a51-plugins-config-volume" pod "309d5f96-0577-45a6-8855-727412314a51" (UID: "309d5f96-0577-45a6-8855-727412314a51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:59:20.033424 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:20.033402    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lvch\" (UniqueName: \"kubernetes.io/projected/db5bfc00-f082-405e-8d43-843cbaf95b52-kube-api-access-8lvch\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:20.035333 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:20.035303    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/309d5f96-0577-45a6-8855-727412314a51-kube-api-access-tvt77" pod "309d5f96-0577-45a6-8855-727412314a51" (UID: "309d5f96-0577-45a6-8855-727412314a51"). InnerVolumeSpecName "kube-api-access-tvt77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:59:20.135606 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:20.135569    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/309d5f96-0577-45a6-8855-727412314a51-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:20.135606 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:20.135607    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvt77\" (UniqueName: \"kubernetes.io/projected/309d5f96-0577-45a6-8855-727412314a51-kube-api-access-tvt77\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:20.213528 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:20.213493    5591 generic.go:356] "Generic (PLEG): container finished" podID="309d5f96-0577-45a6-8855-727412314a51" containerID="53bf3186af373d21a94c9eebf3a15b8964c59391ca81a814976d5f734b895c7a" exitCode=0
Nov 03 19:59:20.213680 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:20.213585    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4" event={"ID":"309d5f96-0577-45a6-8855-727412314a51","Type":"ContainerDied","Data":"53bf3186af373d21a94c9eebf3a15b8964c59391ca81a814976d5f734b895c7a"}
Nov 03 19:59:20.213680 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:20.213595    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4"
Nov 03 19:59:20.213680 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:20.213633    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4" event={"ID":"309d5f96-0577-45a6-8855-727412314a51","Type":"ContainerDied","Data":"0fa3554d6c21f7a8eca61e9201bd86c82537097c6f587eea2a85808d82b78bc1"}
Nov 03 19:59:20.213680 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:20.213649    5591 scope.go:122] "RemoveContainer" containerID="53bf3186af373d21a94c9eebf3a15b8964c59391ca81a814976d5f734b895c7a"
Nov 03 19:59:20.215288 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:20.215267    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh"
Nov 03 19:59:20.215288 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:20.215197    5591 generic.go:356] "Generic (PLEG): container finished" podID="db5bfc00-f082-405e-8d43-843cbaf95b52" containerID="62f7076414d6d48e66c1196cca0beddf60c3285d705c1e48dec5cf9e17b4c7c2" exitCode=0
Nov 03 19:59:20.215483 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:20.215299    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh" event={"ID":"db5bfc00-f082-405e-8d43-843cbaf95b52","Type":"ContainerDied","Data":"62f7076414d6d48e66c1196cca0beddf60c3285d705c1e48dec5cf9e17b4c7c2"}
Nov 03 19:59:20.215483 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:20.215344    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh" event={"ID":"db5bfc00-f082-405e-8d43-843cbaf95b52","Type":"ContainerDied","Data":"c2deafa393be9fbbcfb0cee63b2b780bfa982cd1e57d335d394bf51198d3667b"}
Nov 03 19:59:20.226918 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:20.226871    5591 scope.go:122] "RemoveContainer" containerID="53bf3186af373d21a94c9eebf3a15b8964c59391ca81a814976d5f734b895c7a"
Nov 03 19:59:20.227290 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:59:20.227264    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53bf3186af373d21a94c9eebf3a15b8964c59391ca81a814976d5f734b895c7a\": container with ID starting with 53bf3186af373d21a94c9eebf3a15b8964c59391ca81a814976d5f734b895c7a not found: ID does not exist" containerID="53bf3186af373d21a94c9eebf3a15b8964c59391ca81a814976d5f734b895c7a"
Nov 03 19:59:20.227392 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:20.227299    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53bf3186af373d21a94c9eebf3a15b8964c59391ca81a814976d5f734b895c7a"} err="failed to get container status \"53bf3186af373d21a94c9eebf3a15b8964c59391ca81a814976d5f734b895c7a\": rpc error: code = NotFound desc = could not find container \"53bf3186af373d21a94c9eebf3a15b8964c59391ca81a814976d5f734b895c7a\": container with ID starting with 53bf3186af373d21a94c9eebf3a15b8964c59391ca81a814976d5f734b895c7a not found: ID does not exist"
Nov 03 19:59:20.227392 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:20.227318    5591 scope.go:122] "RemoveContainer" containerID="62f7076414d6d48e66c1196cca0beddf60c3285d705c1e48dec5cf9e17b4c7c2"
Nov 03 19:59:20.239601 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:20.239556    5591 scope.go:122] "RemoveContainer" containerID="62f7076414d6d48e66c1196cca0beddf60c3285d705c1e48dec5cf9e17b4c7c2"
Nov 03 19:59:20.239973 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:59:20.239931    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62f7076414d6d48e66c1196cca0beddf60c3285d705c1e48dec5cf9e17b4c7c2\": container with ID starting with 62f7076414d6d48e66c1196cca0beddf60c3285d705c1e48dec5cf9e17b4c7c2 not found: ID does not exist" containerID="62f7076414d6d48e66c1196cca0beddf60c3285d705c1e48dec5cf9e17b4c7c2"
Nov 03 19:59:20.240098 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:20.239965    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f7076414d6d48e66c1196cca0beddf60c3285d705c1e48dec5cf9e17b4c7c2"} err="failed to get container status \"62f7076414d6d48e66c1196cca0beddf60c3285d705c1e48dec5cf9e17b4c7c2\": rpc error: code = NotFound desc = could not find container \"62f7076414d6d48e66c1196cca0beddf60c3285d705c1e48dec5cf9e17b4c7c2\": container with ID starting with 62f7076414d6d48e66c1196cca0beddf60c3285d705c1e48dec5cf9e17b4c7c2 not found: ID does not exist"
Nov 03 19:59:20.243299 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:20.243267    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4"]
Nov 03 19:59:20.248896 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:20.248870    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-277dc9-9d5658d5f-7wxj4"]
Nov 03 19:59:20.265625 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:20.265592    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh"]
Nov 03 19:59:20.281832 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:20.281789    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-277dc9-59bfb9d7f8-kl2kh"]
Nov 03 19:59:21.152078 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:21.152042    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f"]
Nov 03 19:59:21.153052 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:21.153028    5591 cpu_manager.go:414] "containerMap: removing container" podUID="db5bfc00-f082-405e-8d43-843cbaf95b52" containerName="payload-processing"
Nov 03 19:59:21.153052 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:21.153051    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db5bfc00-f082-405e-8d43-843cbaf95b52" containerName="payload-processing"
Nov 03 19:59:21.153279 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:21.153064    5591 cpu_manager.go:414] "containerMap: removing container" podUID="309d5f96-0577-45a6-8855-727412314a51" containerName="payload-pre-processing"
Nov 03 19:59:21.153279 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:21.153071    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="309d5f96-0577-45a6-8855-727412314a51" containerName="payload-pre-processing"
Nov 03 19:59:21.153279 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:21.153276    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="db5bfc00-f082-405e-8d43-843cbaf95b52" containerName="payload-processing"
Nov 03 19:59:21.153440 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:21.153293    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="309d5f96-0577-45a6-8855-727412314a51" containerName="payload-pre-processing"
Nov 03 19:59:21.163901 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:21.163863    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f"]
Nov 03 19:59:21.164063 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:21.164022    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f"
Nov 03 19:59:21.168055 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:21.168028    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c6092e-serving-cert\""
Nov 03 19:59:21.305641 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:21.273143    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4035d2b1-0439-40e5-a01a-209194d91356-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f\" (UID: \"4035d2b1-0439-40e5-a01a-209194d91356\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f"
Nov 03 19:59:21.306004 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:21.305972    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4035d2b1-0439-40e5-a01a-209194d91356-metrics-certs\") pod \"maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f\" (UID: \"4035d2b1-0439-40e5-a01a-209194d91356\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f"
Nov 03 19:59:21.325358 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:21.325317    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4035d2b1-0439-40e5-a01a-209194d91356-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f\" (UID: \"4035d2b1-0439-40e5-a01a-209194d91356\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f"
Nov 03 19:59:21.325686 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:21.325663    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f49jr\" (UniqueName: \"kubernetes.io/projected/4035d2b1-0439-40e5-a01a-209194d91356-kube-api-access-f49jr\") pod \"maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f\" (UID: \"4035d2b1-0439-40e5-a01a-209194d91356\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f"
Nov 03 19:59:21.388253 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:21.387800    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="309d5f96-0577-45a6-8855-727412314a51" path="/var/lib/kubelet/pods/309d5f96-0577-45a6-8855-727412314a51/volumes"
Nov 03 19:59:21.388462 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:21.388328    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db5bfc00-f082-405e-8d43-843cbaf95b52" path="/var/lib/kubelet/pods/db5bfc00-f082-405e-8d43-843cbaf95b52/volumes"
Nov 03 19:59:21.426983 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:21.426884    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4035d2b1-0439-40e5-a01a-209194d91356-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f\" (UID: \"4035d2b1-0439-40e5-a01a-209194d91356\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f"
Nov 03 19:59:21.426983 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:21.426961    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f49jr\" (UniqueName: \"kubernetes.io/projected/4035d2b1-0439-40e5-a01a-209194d91356-kube-api-access-f49jr\") pod \"maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f\" (UID: \"4035d2b1-0439-40e5-a01a-209194d91356\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f"
Nov 03 19:59:21.427259 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:21.427141    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4035d2b1-0439-40e5-a01a-209194d91356-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f\" (UID: \"4035d2b1-0439-40e5-a01a-209194d91356\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f"
Nov 03 19:59:21.427259 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:59:21.427240    5591 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-serving-cert: secret "maas-api-e2e-ipp-a-c6092e-serving-cert" not found
Nov 03 19:59:21.427440 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:59:21.427312    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4035d2b1-0439-40e5-a01a-209194d91356-maas-api-tls podName:4035d2b1-0439-40e5-a01a-209194d91356 nodeName:}" failed. No retries permitted until 2026-11-03 19:59:21.927290385 +0000 UTC m=+1310.699583381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4035d2b1-0439-40e5-a01a-209194d91356-maas-api-tls") pod "maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f" (UID: "4035d2b1-0439-40e5-a01a-209194d91356") : secret "maas-api-e2e-ipp-a-c6092e-serving-cert" not found
Nov 03 19:59:21.427440 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:21.427393    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4035d2b1-0439-40e5-a01a-209194d91356-metrics-certs\") pod \"maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f\" (UID: \"4035d2b1-0439-40e5-a01a-209194d91356\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f"
Nov 03 19:59:21.427766 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:21.427746    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4035d2b1-0439-40e5-a01a-209194d91356-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f\" (UID: \"4035d2b1-0439-40e5-a01a-209194d91356\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f"
Nov 03 19:59:21.429891 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:21.429873    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4035d2b1-0439-40e5-a01a-209194d91356-metrics-certs\") pod \"maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f\" (UID: \"4035d2b1-0439-40e5-a01a-209194d91356\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f"
Nov 03 19:59:21.442600 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:21.442579    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f49jr\" (UniqueName: \"kubernetes.io/projected/4035d2b1-0439-40e5-a01a-209194d91356-kube-api-access-f49jr\") pod \"maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f\" (UID: \"4035d2b1-0439-40e5-a01a-209194d91356\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f"
Nov 03 19:59:21.940708 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:21.940634    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4035d2b1-0439-40e5-a01a-209194d91356-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f\" (UID: \"4035d2b1-0439-40e5-a01a-209194d91356\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f"
Nov 03 19:59:21.968194 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:21.968162    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4035d2b1-0439-40e5-a01a-209194d91356-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f\" (UID: \"4035d2b1-0439-40e5-a01a-209194d91356\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f"
Nov 03 19:59:22.082902 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:22.082871    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f"
Nov 03 19:59:22.540013 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:22.539979    5591 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:59:22.550846 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:22.550812    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f"]
Nov 03 19:59:22.956827 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:22.956797    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:59:23.058648 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.058607    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3377787a-22d0-45ce-9c24-c39c840af373-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3377787a-22d0-45ce-9c24-c39c840af373-istiod-ca-cert\") pod \"3377787a-22d0-45ce-9c24-c39c840af373\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") "
Nov 03 19:59:23.058830 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.058707    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-workload-certs\") pod \"3377787a-22d0-45ce-9c24-c39c840af373\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") "
Nov 03 19:59:23.058894 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.058867    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-credential-socket\") pod \"3377787a-22d0-45ce-9c24-c39c840af373\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") "
Nov 03 19:59:23.058944 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.058906    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3377787a-22d0-45ce-9c24-c39c840af373-istio-token\" (UniqueName: \"kubernetes.io/projected/3377787a-22d0-45ce-9c24-c39c840af373-istio-token\") pod \"3377787a-22d0-45ce-9c24-c39c840af373\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") "
Nov 03 19:59:23.058981 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.058940    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-workload-socket\") pod \"3377787a-22d0-45ce-9c24-c39c840af373\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") "
Nov 03 19:59:23.059050 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.059015    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-workload-certs" pod "3377787a-22d0-45ce-9c24-c39c840af373" (UID: "3377787a-22d0-45ce-9c24-c39c840af373"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:59:23.059123 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.059070    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-istio-data\") pod \"3377787a-22d0-45ce-9c24-c39c840af373\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") "
Nov 03 19:59:23.059123 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.059098    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3377787a-22d0-45ce-9c24-c39c840af373-istiod-ca-cert" pod "3377787a-22d0-45ce-9c24-c39c840af373" (UID: "3377787a-22d0-45ce-9c24-c39c840af373"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:59:23.059275 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.059137    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-istio-envoy\") pod \"3377787a-22d0-45ce-9c24-c39c840af373\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") "
Nov 03 19:59:23.059275 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.059156    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-credential-socket" pod "3377787a-22d0-45ce-9c24-c39c840af373" (UID: "3377787a-22d0-45ce-9c24-c39c840af373"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:59:23.059275 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.059174    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3377787a-22d0-45ce-9c24-c39c840af373-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3377787a-22d0-45ce-9c24-c39c840af373-istio-podinfo\") pod \"3377787a-22d0-45ce-9c24-c39c840af373\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") "
Nov 03 19:59:23.059275 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.059242    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3377787a-22d0-45ce-9c24-c39c840af373-kube-api-access-766zv\" (UniqueName: \"kubernetes.io/projected/3377787a-22d0-45ce-9c24-c39c840af373-kube-api-access-766zv\") pod \"3377787a-22d0-45ce-9c24-c39c840af373\" (UID: \"3377787a-22d0-45ce-9c24-c39c840af373\") "
Nov 03 19:59:23.059512 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.059324    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-workload-socket" pod "3377787a-22d0-45ce-9c24-c39c840af373" (UID: "3377787a-22d0-45ce-9c24-c39c840af373"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:59:23.059512 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.059392    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-istio-data" pod "3377787a-22d0-45ce-9c24-c39c840af373" (UID: "3377787a-22d0-45ce-9c24-c39c840af373"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:59:23.059902 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.059873    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:23.059902 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.059897    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3377787a-22d0-45ce-9c24-c39c840af373-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:23.060076 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.059911    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:23.060076 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.059924    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:23.060076 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.059936    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:23.061739 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.061719    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3377787a-22d0-45ce-9c24-c39c840af373-istio-podinfo" pod "3377787a-22d0-45ce-9c24-c39c840af373" (UID: "3377787a-22d0-45ce-9c24-c39c840af373"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:59:23.061915 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.061901    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-istio-envoy" pod "3377787a-22d0-45ce-9c24-c39c840af373" (UID: "3377787a-22d0-45ce-9c24-c39c840af373"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:59:23.062046 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.062027    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3377787a-22d0-45ce-9c24-c39c840af373-kube-api-access-766zv" pod "3377787a-22d0-45ce-9c24-c39c840af373" (UID: "3377787a-22d0-45ce-9c24-c39c840af373"). InnerVolumeSpecName "kube-api-access-766zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:59:23.062102 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.062074    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3377787a-22d0-45ce-9c24-c39c840af373-istio-token" pod "3377787a-22d0-45ce-9c24-c39c840af373" (UID: "3377787a-22d0-45ce-9c24-c39c840af373"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:59:23.165230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.164610    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3377787a-22d0-45ce-9c24-c39c840af373-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:23.165230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.164662    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3377787a-22d0-45ce-9c24-c39c840af373-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:23.165230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.164678    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3377787a-22d0-45ce-9c24-c39c840af373-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:23.165230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.164694    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-766zv\" (UniqueName: \"kubernetes.io/projected/3377787a-22d0-45ce-9c24-c39c840af373-kube-api-access-766zv\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:23.419699 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.419656    5591 generic.go:356] "Generic (PLEG): container finished" podID="770df5ab-84a7-4fe9-999b-8ca9ee6464a4" containerID="05986903284faf908de90f1f43aab54545c06a3e029af3a54a2d6c3731028307" exitCode=0
Nov 03 19:59:23.421169 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.421142    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 19:59:23.422302 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.422271    5591 generic.go:356] "Generic (PLEG): container finished" podID="3377787a-22d0-45ce-9c24-c39c840af373" containerID="d07698bd0f99375147174643e3f8fd9636b15d9f5f0e389a08272a36c44231eb" exitCode=0
Nov 03 19:59:23.422821 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.422672    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"
Nov 03 19:59:23.426575 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.424789    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc" event={"ID":"770df5ab-84a7-4fe9-999b-8ca9ee6464a4","Type":"ContainerDied","Data":"05986903284faf908de90f1f43aab54545c06a3e029af3a54a2d6c3731028307"}
Nov 03 19:59:23.426575 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.424852    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds" event={"ID":"3377787a-22d0-45ce-9c24-c39c840af373","Type":"ContainerDied","Data":"d07698bd0f99375147174643e3f8fd9636b15d9f5f0e389a08272a36c44231eb"}
Nov 03 19:59:23.426575 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.424880    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds" event={"ID":"3377787a-22d0-45ce-9c24-c39c840af373","Type":"ContainerDied","Data":"5ad5c5030029d1a366dce8ba417c0b55ee2bfdbeadc97ebe3be8b6fb93ad51db"}
Nov 03 19:59:23.426575 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.424910    5591 scope.go:122] "RemoveContainer" containerID="d07698bd0f99375147174643e3f8fd9636b15d9f5f0e389a08272a36c44231eb"
Nov 03 19:59:23.445976 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.445934    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f" event={"ID":"4035d2b1-0439-40e5-a01a-209194d91356","Type":"ContainerStarted","Data":"ba1fb2ea352c28a5e23829b813f3cd7a493aa3446c2b104040bcf65f559535c9"}
Nov 03 19:59:23.460431 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.460398    5591 scope.go:122] "RemoveContainer" containerID="d07698bd0f99375147174643e3f8fd9636b15d9f5f0e389a08272a36c44231eb"
Nov 03 19:59:23.461098 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:59:23.461068    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d07698bd0f99375147174643e3f8fd9636b15d9f5f0e389a08272a36c44231eb\": container with ID starting with d07698bd0f99375147174643e3f8fd9636b15d9f5f0e389a08272a36c44231eb not found: ID does not exist" containerID="d07698bd0f99375147174643e3f8fd9636b15d9f5f0e389a08272a36c44231eb"
Nov 03 19:59:23.461339 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.461109    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d07698bd0f99375147174643e3f8fd9636b15d9f5f0e389a08272a36c44231eb"} err="failed to get container status \"d07698bd0f99375147174643e3f8fd9636b15d9f5f0e389a08272a36c44231eb\": rpc error: code = NotFound desc = could not find container \"d07698bd0f99375147174643e3f8fd9636b15d9f5f0e389a08272a36c44231eb\": container with ID starting with d07698bd0f99375147174643e3f8fd9636b15d9f5f0e389a08272a36c44231eb not found: ID does not exist"
Nov 03 19:59:23.469505 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.467528    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"]
Nov 03 19:59:23.474521 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.474487    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-l2rds"]
Nov 03 19:59:23.700282 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.698599    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:59:23.801226 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.800706    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-podinfo\") pod \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") "
Nov 03 19:59:23.801226 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.800761    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-token\" (UniqueName: \"kubernetes.io/projected/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-token\") pod \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") "
Nov 03 19:59:23.801226 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.800934    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-workload-socket\") pod \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") "
Nov 03 19:59:23.801226 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.800969    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-data\") pod \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") "
Nov 03 19:59:23.801226 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.801063    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-credential-socket\") pod \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") "
Nov 03 19:59:23.801226 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.801160    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istiod-ca-cert\") pod \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") "
Nov 03 19:59:23.801226 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.801188    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-workload-certs\") pod \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") "
Nov 03 19:59:23.801781 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.801249    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-kube-api-access-b7tw5\" (UniqueName: \"kubernetes.io/projected/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-kube-api-access-b7tw5\") pod \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") "
Nov 03 19:59:23.801781 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.801284    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-envoy\") pod \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\" (UID: \"770df5ab-84a7-4fe9-999b-8ca9ee6464a4\") "
Nov 03 19:59:23.834290 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.816126    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-data" pod "770df5ab-84a7-4fe9-999b-8ca9ee6464a4" (UID: "770df5ab-84a7-4fe9-999b-8ca9ee6464a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:59:23.864836 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.844542    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-workload-socket" pod "770df5ab-84a7-4fe9-999b-8ca9ee6464a4" (UID: "770df5ab-84a7-4fe9-999b-8ca9ee6464a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:59:23.864836 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.844726    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-credential-socket" pod "770df5ab-84a7-4fe9-999b-8ca9ee6464a4" (UID: "770df5ab-84a7-4fe9-999b-8ca9ee6464a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:59:23.864836 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.845454    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-workload-certs" pod "770df5ab-84a7-4fe9-999b-8ca9ee6464a4" (UID: "770df5ab-84a7-4fe9-999b-8ca9ee6464a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:59:23.864836 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.845964    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-kube-api-access-b7tw5" pod "770df5ab-84a7-4fe9-999b-8ca9ee6464a4" (UID: "770df5ab-84a7-4fe9-999b-8ca9ee6464a4"). InnerVolumeSpecName "kube-api-access-b7tw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:59:23.865733 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.865697    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istiod-ca-cert" pod "770df5ab-84a7-4fe9-999b-8ca9ee6464a4" (UID: "770df5ab-84a7-4fe9-999b-8ca9ee6464a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:59:23.893173 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.892919    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-token" pod "770df5ab-84a7-4fe9-999b-8ca9ee6464a4" (UID: "770df5ab-84a7-4fe9-999b-8ca9ee6464a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:59:23.901381 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.901343    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-podinfo" pod "770df5ab-84a7-4fe9-999b-8ca9ee6464a4" (UID: "770df5ab-84a7-4fe9-999b-8ca9ee6464a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:59:23.901849 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.901819    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-envoy" pod "770df5ab-84a7-4fe9-999b-8ca9ee6464a4" (UID: "770df5ab-84a7-4fe9-999b-8ca9ee6464a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:59:23.903943 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.903823    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:23.903943 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.903845    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:23.903943 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.903855    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:23.903943 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.903864    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:23.903943 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.903874    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:23.904177 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.903883    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:23.904177 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.904034    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:23.904177 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.904044    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7tw5\" (UniqueName: \"kubernetes.io/projected/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-kube-api-access-b7tw5\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:23.904177 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:23.904054    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/770df5ab-84a7-4fe9-999b-8ca9ee6464a4-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:24.454469 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:24.454437    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"
Nov 03 19:59:24.454658 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:24.454443    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc" event={"ID":"770df5ab-84a7-4fe9-999b-8ca9ee6464a4","Type":"ContainerDied","Data":"19c02d52dd3fa71fb4ffd28939b23f5c55cb84988ad07176b125d66ff2b3be00"}
Nov 03 19:59:24.454658 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:24.454594    5591 scope.go:122] "RemoveContainer" containerID="05986903284faf908de90f1f43aab54545c06a3e029af3a54a2d6c3731028307"
Nov 03 19:59:24.458094 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:24.458051    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f" event={"ID":"4035d2b1-0439-40e5-a01a-209194d91356","Type":"ContainerStarted","Data":"f2ed0f056de49d02807bdde29cfe7e6a26e9e16a70fd3598760974f411398e36"}
Nov 03 19:59:24.458278 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:24.458260    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f"
Nov 03 19:59:24.483838 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:24.483789    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f" podStartSLOduration=2.949778438 podStartE2EDuration="3.48377332s" totalImagesPullingTime="533.994882ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:24.480236445 +0000 UTC m=+1313.252529466" watchObservedRunningTime="2026-11-03 19:59:24.48377332 +0000 UTC m=+1313.256066345"
Nov 03 19:59:24.505299 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:24.505261    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"]
Nov 03 19:59:24.510160 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:24.509896    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-978qc"]
Nov 03 19:59:25.159110 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:25.159081    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw"
Nov 03 19:59:25.159513 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:25.159151    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx"
Nov 03 19:59:25.351875 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:25.351840    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3377787a-22d0-45ce-9c24-c39c840af373" path="/var/lib/kubelet/pods/3377787a-22d0-45ce-9c24-c39c840af373/volumes"
Nov 03 19:59:25.352528 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:25.352506    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="770df5ab-84a7-4fe9-999b-8ca9ee6464a4" path="/var/lib/kubelet/pods/770df5ab-84a7-4fe9-999b-8ca9ee6464a4/volumes"
Nov 03 19:59:28.728319 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:59:28.728282    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:59:30.530745 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:30.530713    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f"
Nov 03 19:59:30.830632 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:30.830565    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-x84lg_d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d/payload-processing/0.log"
Nov 03 19:59:31.072256 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:31.070515    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-x84lg_d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d/payload-processing/0.log"
Nov 03 19:59:33.048655 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.048613    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"]
Nov 03 19:59:33.049858 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.049828    5591 cpu_manager.go:414] "containerMap: removing container" podUID="770df5ab-84a7-4fe9-999b-8ca9ee6464a4" containerName="istio-proxy"
Nov 03 19:59:33.049858 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.049855    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="770df5ab-84a7-4fe9-999b-8ca9ee6464a4" containerName="istio-proxy"
Nov 03 19:59:33.050174 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.049878    5591 cpu_manager.go:414] "containerMap: removing container" podUID="3377787a-22d0-45ce-9c24-c39c840af373" containerName="istio-proxy"
Nov 03 19:59:33.050174 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.049886    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3377787a-22d0-45ce-9c24-c39c840af373" containerName="istio-proxy"
Nov 03 19:59:33.050174 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.050089    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="770df5ab-84a7-4fe9-999b-8ca9ee6464a4" containerName="istio-proxy"
Nov 03 19:59:33.050174 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.050103    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3377787a-22d0-45ce-9c24-c39c840af373" containerName="istio-proxy"
Nov 03 19:59:33.056711 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.056689    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.064401 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.064374    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"]
Nov 03 19:59:33.066605 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.065944    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-5cf30a-openshift-default-dockercfg-7hx56\""
Nov 03 19:59:33.106579 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.106533    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-credential-socket\") pod \"e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") " pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.106779 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.106587    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-envoy\") pod \"e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") " pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.106779 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.106621    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-workload-socket\") pod \"e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") " pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.106953 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.106924    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/928410e8-ddb8-4a61-8a41-f218603c0e47-istiod-ca-cert\") pod \"e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") " pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.107048 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.107033    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-workload-certs\") pod \"e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") " pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.107128 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.107115    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-podinfo\") pod \"e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") " pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.107184 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.107153    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-token\") pod \"e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") " pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.107382 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.107356    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-data\") pod \"e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") " pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.107438 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.107415    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzrcb\" (UniqueName: \"kubernetes.io/projected/928410e8-ddb8-4a61-8a41-f218603c0e47-kube-api-access-gzrcb\") pod \"e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") " pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.218231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.209374    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-credential-socket\") pod \"e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") " pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.218231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.209428    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-envoy\") pod \"e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") " pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.218231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.209457    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-workload-socket\") pod \"e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") " pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.218231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.209545    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/928410e8-ddb8-4a61-8a41-f218603c0e47-istiod-ca-cert\") pod \"e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") " pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.218231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.209634    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-workload-certs\") pod \"e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") " pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.218231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.209712    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-podinfo\") pod \"e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") " pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.218231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.209747    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-token\") pod \"e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") " pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.218231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.209800    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-data\") pod \"e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") " pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.218231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.209841    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzrcb\" (UniqueName: \"kubernetes.io/projected/928410e8-ddb8-4a61-8a41-f218603c0e47-kube-api-access-gzrcb\") pod \"e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") " pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.218231 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.211292    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-credential-socket\") pod \"e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") " pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.219364 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.219330    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-workload-certs\") pod \"e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") " pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.219777 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.219757    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-workload-socket\") pod \"e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") " pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.232602 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.232568    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/928410e8-ddb8-4a61-8a41-f218603c0e47-istiod-ca-cert\") pod \"e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") " pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.250228 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.232935    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-data\") pod \"e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") " pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.272233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.258103    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-envoy\") pod \"e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") " pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.272233 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.271941    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzrcb\" (UniqueName: \"kubernetes.io/projected/928410e8-ddb8-4a61-8a41-f218603c0e47-kube-api-access-gzrcb\") pod \"e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") " pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.286696 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.278671    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-token\") pod \"e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") " pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.293272 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.292560    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-podinfo\") pod \"e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") " pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.387879 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.387797    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:33.753077 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:59:33.753045    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod928410e8_ddb8_4a61_8a41_f218603c0e47.slice/crio-e15c6b3ed45cb8bdc07eedf23c13decfa7343203ce9f6f6475fc49c7e7901c84 WatchSource:0}: Error finding container e15c6b3ed45cb8bdc07eedf23c13decfa7343203ce9f6f6475fc49c7e7901c84: Status 404 returned error can't find the container with id e15c6b3ed45cb8bdc07eedf23c13decfa7343203ce9f6f6475fc49c7e7901c84
Nov 03 19:59:33.753198 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.753153    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"]
Nov 03 19:59:33.755760 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.755730    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:59:33.755817 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.755793    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:59:33.755856 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:33.755824    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 19:59:34.670256 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:34.670193    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz" event={"ID":"928410e8-ddb8-4a61-8a41-f218603c0e47","Type":"ContainerStarted","Data":"51ca72b4a5ae046f96e4fd65781a3cd4a97f7078c6ab97ca0415efd732c3f754"}
Nov 03 19:59:34.670749 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:34.670279    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz" event={"ID":"928410e8-ddb8-4a61-8a41-f218603c0e47","Type":"ContainerStarted","Data":"e15c6b3ed45cb8bdc07eedf23c13decfa7343203ce9f6f6475fc49c7e7901c84"}
Nov 03 19:59:34.709074 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:34.709012    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz" podStartSLOduration=1.708992507 podStartE2EDuration="1.708992507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:34.706048931 +0000 UTC m=+1323.478341948" watchObservedRunningTime="2026-11-03 19:59:34.708992507 +0000 UTC m=+1323.481285526"
Nov 03 19:59:35.393345 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:35.393313    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:35.400039 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:35.399995    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:35.596230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:35.594770    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw"]
Nov 03 19:59:35.894287 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:35.890130    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx"]
Nov 03 19:59:36.012007 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:36.011971    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:36.016226 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:36.016180    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 19:59:38.457961 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:38.457931    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6"]
Nov 03 19:59:38.462585 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:38.462555    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6"
Nov 03 19:59:38.472602 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:38.472576    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6"]
Nov 03 19:59:38.473556 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:38.473529    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-5cf30a\""
Nov 03 19:59:38.476270 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:38.476249    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-5cf30a-dockercfg-r22sd\""
Nov 03 19:59:38.486745 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:38.486716    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmnnc\" (UniqueName: \"kubernetes.io/projected/43ad69e4-a321-4324-9e2a-9789fed758e9-kube-api-access-kmnnc\") pod \"payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6\" (UID: \"43ad69e4-a321-4324-9e2a-9789fed758e9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6"
Nov 03 19:59:38.486913 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:38.486803    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ad69e4-a321-4324-9e2a-9789fed758e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6\" (UID: \"43ad69e4-a321-4324-9e2a-9789fed758e9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6"
Nov 03 19:59:38.500365 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:38.500333    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf"]
Nov 03 19:59:38.505114 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:38.505090    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf"
Nov 03 19:59:38.513963 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:38.513938    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf"]
Nov 03 19:59:38.704539 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:38.702891    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmnnc\" (UniqueName: \"kubernetes.io/projected/43ad69e4-a321-4324-9e2a-9789fed758e9-kube-api-access-kmnnc\") pod \"payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6\" (UID: \"43ad69e4-a321-4324-9e2a-9789fed758e9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6"
Nov 03 19:59:38.704539 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:38.703016    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ad69e4-a321-4324-9e2a-9789fed758e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6\" (UID: \"43ad69e4-a321-4324-9e2a-9789fed758e9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6"
Nov 03 19:59:38.704539 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:38.703117    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf2vl\" (UniqueName: \"kubernetes.io/projected/3efe99ef-4e27-4f9c-994b-ebdd23792b8c-kube-api-access-lf2vl\") pod \"payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf\" (UID: \"3efe99ef-4e27-4f9c-994b-ebdd23792b8c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf"
Nov 03 19:59:38.704539 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:38.703191    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3efe99ef-4e27-4f9c-994b-ebdd23792b8c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf\" (UID: \"3efe99ef-4e27-4f9c-994b-ebdd23792b8c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf"
Nov 03 19:59:38.704539 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:38.704488    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ad69e4-a321-4324-9e2a-9789fed758e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6\" (UID: \"43ad69e4-a321-4324-9e2a-9789fed758e9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6"
Nov 03 19:59:38.765326 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:38.763942    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmnnc\" (UniqueName: \"kubernetes.io/projected/43ad69e4-a321-4324-9e2a-9789fed758e9-kube-api-access-kmnnc\") pod \"payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6\" (UID: \"43ad69e4-a321-4324-9e2a-9789fed758e9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6"
Nov 03 19:59:38.789434 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:38.789042    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6"
Nov 03 19:59:38.812861 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:38.811672    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf2vl\" (UniqueName: \"kubernetes.io/projected/3efe99ef-4e27-4f9c-994b-ebdd23792b8c-kube-api-access-lf2vl\") pod \"payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf\" (UID: \"3efe99ef-4e27-4f9c-994b-ebdd23792b8c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf"
Nov 03 19:59:38.830524 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:38.830474    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3efe99ef-4e27-4f9c-994b-ebdd23792b8c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf\" (UID: \"3efe99ef-4e27-4f9c-994b-ebdd23792b8c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf"
Nov 03 19:59:38.839974 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:38.831598    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3efe99ef-4e27-4f9c-994b-ebdd23792b8c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf\" (UID: \"3efe99ef-4e27-4f9c-994b-ebdd23792b8c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf"
Nov 03 19:59:38.851175 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:38.851140    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf2vl\" (UniqueName: \"kubernetes.io/projected/3efe99ef-4e27-4f9c-994b-ebdd23792b8c-kube-api-access-lf2vl\") pod \"payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf\" (UID: \"3efe99ef-4e27-4f9c-994b-ebdd23792b8c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf"
Nov 03 19:59:39.124248 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:39.124119    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf"
Nov 03 19:59:39.159633 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:39.159601    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6"]
Nov 03 19:59:39.160915 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:59:39.160871    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ad69e4_a321_4324_9e2a_9789fed758e9.slice/crio-a7c9eac56d1472bc578b99577b1900cfdb8ad992852c4e8933ce3f2be891bf92 WatchSource:0}: Error finding container a7c9eac56d1472bc578b99577b1900cfdb8ad992852c4e8933ce3f2be891bf92: Status 404 returned error can't find the container with id a7c9eac56d1472bc578b99577b1900cfdb8ad992852c4e8933ce3f2be891bf92
Nov 03 19:59:39.302090 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:39.302058    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-x84lg_d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d/payload-processing/0.log"
Nov 03 19:59:39.422187 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:39.422157    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf"]
Nov 03 19:59:39.423033 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:59:39.423008    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3efe99ef_4e27_4f9c_994b_ebdd23792b8c.slice/crio-1868c3eeb00e73e0e736ab2e2cbf62c55f2ea53cebbf3685f4ed931c1ad07458 WatchSource:0}: Error finding container 1868c3eeb00e73e0e736ab2e2cbf62c55f2ea53cebbf3685f4ed931c1ad07458: Status 404 returned error can't find the container with id 1868c3eeb00e73e0e736ab2e2cbf62c55f2ea53cebbf3685f4ed931c1ad07458
Nov 03 19:59:39.484394 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:39.484374    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-x84lg_d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d/payload-processing/0.log"
Nov 03 19:59:40.050759 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:40.050712    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf" event={"ID":"3efe99ef-4e27-4f9c-994b-ebdd23792b8c","Type":"ContainerStarted","Data":"5909ab93566cf248a5770b3b87c73c397def596e09635a91376a42fb74862800"}
Nov 03 19:59:40.052553 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:40.051019    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf" event={"ID":"3efe99ef-4e27-4f9c-994b-ebdd23792b8c","Type":"ContainerStarted","Data":"1868c3eeb00e73e0e736ab2e2cbf62c55f2ea53cebbf3685f4ed931c1ad07458"}
Nov 03 19:59:40.052553 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:40.051137    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf"
Nov 03 19:59:40.054658 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:40.054626    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6" event={"ID":"43ad69e4-a321-4324-9e2a-9789fed758e9","Type":"ContainerStarted","Data":"14fca4a8d55eda6e007e9c889f97fa769ad47a63ee7a876071c46fe658103c47"}
Nov 03 19:59:40.054809 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:40.054682    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6" event={"ID":"43ad69e4-a321-4324-9e2a-9789fed758e9","Type":"ContainerStarted","Data":"a7c9eac56d1472bc578b99577b1900cfdb8ad992852c4e8933ce3f2be891bf92"}
Nov 03 19:59:40.055390 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:40.055366    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6"
Nov 03 19:59:40.141281 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:40.141223    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf" podStartSLOduration=2.14118422 podStartE2EDuration="2.14118422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:40.139159632 +0000 UTC m=+1328.911452651" watchObservedRunningTime="2026-11-03 19:59:40.14118422 +0000 UTC m=+1328.913477238"
Nov 03 19:59:40.168555 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:40.168491    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6" podStartSLOduration=2.168471676 podStartE2EDuration="2.168471676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:40.162752351 +0000 UTC m=+1328.935045372" watchObservedRunningTime="2026-11-03 19:59:40.168471676 +0000 UTC m=+1328.940764698"
Nov 03 19:59:40.595803 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:40.595750    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw" podUID="8656a4cc-537b-47c7-9429-6581fb9ab565" containerName="payload-processing" containerID="cri-o://181a37cbef8b4250153447fa2770ae4479375d0b2b7a21db97d31cce259f0048" gracePeriod=25
Nov 03 19:59:40.898452 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:40.898417    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw"
Nov 03 19:59:40.996964 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:40.996924    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx" podUID="5522a57b-9892-4cd8-849e-1c68b47ea721" containerName="payload-pre-processing" containerID="cri-o://7cff7ce7400f0df20e3ce15481b48c1a624a58a22da264124f8f3691e0e34d5f" gracePeriod=25
Nov 03 19:59:41.058404 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:41.058370    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8656a4cc-537b-47c7-9429-6581fb9ab565-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8656a4cc-537b-47c7-9429-6581fb9ab565-plugins-config-volume\") pod \"8656a4cc-537b-47c7-9429-6581fb9ab565\" (UID: \"8656a4cc-537b-47c7-9429-6581fb9ab565\") "
Nov 03 19:59:41.058650 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:41.058632    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8656a4cc-537b-47c7-9429-6581fb9ab565-kube-api-access-pbkv8\" (UniqueName: \"kubernetes.io/projected/8656a4cc-537b-47c7-9429-6581fb9ab565-kube-api-access-pbkv8\") pod \"8656a4cc-537b-47c7-9429-6581fb9ab565\" (UID: \"8656a4cc-537b-47c7-9429-6581fb9ab565\") "
Nov 03 19:59:41.058815 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:41.058762    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8656a4cc-537b-47c7-9429-6581fb9ab565-plugins-config-volume" pod "8656a4cc-537b-47c7-9429-6581fb9ab565" (UID: "8656a4cc-537b-47c7-9429-6581fb9ab565"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:59:41.059887 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:41.059850    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8656a4cc-537b-47c7-9429-6581fb9ab565-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:41.061378 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:41.061355    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8656a4cc-537b-47c7-9429-6581fb9ab565-kube-api-access-pbkv8" pod "8656a4cc-537b-47c7-9429-6581fb9ab565" (UID: "8656a4cc-537b-47c7-9429-6581fb9ab565"). InnerVolumeSpecName "kube-api-access-pbkv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:59:41.064604 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:41.064577    5591 generic.go:356] "Generic (PLEG): container finished" podID="8656a4cc-537b-47c7-9429-6581fb9ab565" containerID="181a37cbef8b4250153447fa2770ae4479375d0b2b7a21db97d31cce259f0048" exitCode=0
Nov 03 19:59:41.064718 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:41.064628    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw"
Nov 03 19:59:41.064718 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:41.064657    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw" event={"ID":"8656a4cc-537b-47c7-9429-6581fb9ab565","Type":"ContainerDied","Data":"181a37cbef8b4250153447fa2770ae4479375d0b2b7a21db97d31cce259f0048"}
Nov 03 19:59:41.064839 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:41.064730    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw" event={"ID":"8656a4cc-537b-47c7-9429-6581fb9ab565","Type":"ContainerDied","Data":"6fd38ab523a0c52b3126e0e183b26a22d44a3f60be99875ea412b614fd71d35b"}
Nov 03 19:59:41.064839 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:41.064765    5591 scope.go:122] "RemoveContainer" containerID="181a37cbef8b4250153447fa2770ae4479375d0b2b7a21db97d31cce259f0048"
Nov 03 19:59:41.102448 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:41.102413    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw"]
Nov 03 19:59:41.105963 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:41.105938    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c6092e-cf9665854-xjmhw"]
Nov 03 19:59:41.112624 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:41.112604    5591 scope.go:122] "RemoveContainer" containerID="181a37cbef8b4250153447fa2770ae4479375d0b2b7a21db97d31cce259f0048"
Nov 03 19:59:41.112977 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:59:41.112951    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"181a37cbef8b4250153447fa2770ae4479375d0b2b7a21db97d31cce259f0048\": container with ID starting with 181a37cbef8b4250153447fa2770ae4479375d0b2b7a21db97d31cce259f0048 not found: ID does not exist" containerID="181a37cbef8b4250153447fa2770ae4479375d0b2b7a21db97d31cce259f0048"
Nov 03 19:59:41.113035 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:41.112982    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"181a37cbef8b4250153447fa2770ae4479375d0b2b7a21db97d31cce259f0048"} err="failed to get container status \"181a37cbef8b4250153447fa2770ae4479375d0b2b7a21db97d31cce259f0048\": rpc error: code = NotFound desc = could not find container \"181a37cbef8b4250153447fa2770ae4479375d0b2b7a21db97d31cce259f0048\": container with ID starting with 181a37cbef8b4250153447fa2770ae4479375d0b2b7a21db97d31cce259f0048 not found: ID does not exist"
Nov 03 19:59:41.165226 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:41.162063    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbkv8\" (UniqueName: \"kubernetes.io/projected/8656a4cc-537b-47c7-9429-6581fb9ab565-kube-api-access-pbkv8\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:41.352765 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:41.352735    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8656a4cc-537b-47c7-9429-6581fb9ab565" path="/var/lib/kubelet/pods/8656a4cc-537b-47c7-9429-6581fb9ab565/volumes"
Nov 03 19:59:41.375550 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:41.375489    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx"
Nov 03 19:59:41.465573 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:41.465537    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5522a57b-9892-4cd8-849e-1c68b47ea721-kube-api-access-hsghk\" (UniqueName: \"kubernetes.io/projected/5522a57b-9892-4cd8-849e-1c68b47ea721-kube-api-access-hsghk\") pod \"5522a57b-9892-4cd8-849e-1c68b47ea721\" (UID: \"5522a57b-9892-4cd8-849e-1c68b47ea721\") "
Nov 03 19:59:41.465806 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:41.465788    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5522a57b-9892-4cd8-849e-1c68b47ea721-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5522a57b-9892-4cd8-849e-1c68b47ea721-plugins-config-volume\") pod \"5522a57b-9892-4cd8-849e-1c68b47ea721\" (UID: \"5522a57b-9892-4cd8-849e-1c68b47ea721\") "
Nov 03 19:59:41.466148 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:41.466125    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5522a57b-9892-4cd8-849e-1c68b47ea721-plugins-config-volume" pod "5522a57b-9892-4cd8-849e-1c68b47ea721" (UID: "5522a57b-9892-4cd8-849e-1c68b47ea721"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:59:41.466437 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:41.466422    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5522a57b-9892-4cd8-849e-1c68b47ea721-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:41.467803 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:41.467782    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5522a57b-9892-4cd8-849e-1c68b47ea721-kube-api-access-hsghk" pod "5522a57b-9892-4cd8-849e-1c68b47ea721" (UID: "5522a57b-9892-4cd8-849e-1c68b47ea721"). InnerVolumeSpecName "kube-api-access-hsghk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:59:41.567796 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:41.567766    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsghk\" (UniqueName: \"kubernetes.io/projected/5522a57b-9892-4cd8-849e-1c68b47ea721-kube-api-access-hsghk\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:42.078152 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:42.077542    5591 generic.go:356] "Generic (PLEG): container finished" podID="5522a57b-9892-4cd8-849e-1c68b47ea721" containerID="7cff7ce7400f0df20e3ce15481b48c1a624a58a22da264124f8f3691e0e34d5f" exitCode=0
Nov 03 19:59:42.078152 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:42.077751    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx"
Nov 03 19:59:42.078728 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:42.078531    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx" event={"ID":"5522a57b-9892-4cd8-849e-1c68b47ea721","Type":"ContainerDied","Data":"7cff7ce7400f0df20e3ce15481b48c1a624a58a22da264124f8f3691e0e34d5f"}
Nov 03 19:59:42.078728 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:42.078614    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx" event={"ID":"5522a57b-9892-4cd8-849e-1c68b47ea721","Type":"ContainerDied","Data":"54b023f4518ad30d3dc850cb3f577072ba26202c6b0f91bba0c4fc5acd44dc7c"}
Nov 03 19:59:42.078877 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:42.078672    5591 scope.go:122] "RemoveContainer" containerID="7cff7ce7400f0df20e3ce15481b48c1a624a58a22da264124f8f3691e0e34d5f"
Nov 03 19:59:42.097838 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:42.095550    5591 scope.go:122] "RemoveContainer" containerID="7cff7ce7400f0df20e3ce15481b48c1a624a58a22da264124f8f3691e0e34d5f"
Nov 03 19:59:42.097838 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:59:42.095895    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cff7ce7400f0df20e3ce15481b48c1a624a58a22da264124f8f3691e0e34d5f\": container with ID starting with 7cff7ce7400f0df20e3ce15481b48c1a624a58a22da264124f8f3691e0e34d5f not found: ID does not exist" containerID="7cff7ce7400f0df20e3ce15481b48c1a624a58a22da264124f8f3691e0e34d5f"
Nov 03 19:59:42.097838 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:42.095934    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cff7ce7400f0df20e3ce15481b48c1a624a58a22da264124f8f3691e0e34d5f"} err="failed to get container status \"7cff7ce7400f0df20e3ce15481b48c1a624a58a22da264124f8f3691e0e34d5f\": rpc error: code = NotFound desc = could not find container \"7cff7ce7400f0df20e3ce15481b48c1a624a58a22da264124f8f3691e0e34d5f\": container with ID starting with 7cff7ce7400f0df20e3ce15481b48c1a624a58a22da264124f8f3691e0e34d5f not found: ID does not exist"
Nov 03 19:59:42.111434 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:42.111407    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx"]
Nov 03 19:59:42.117561 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:42.117534    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-5545cf6799-jdmzx"]
Nov 03 19:59:43.364937 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:43.364904    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5522a57b-9892-4cd8-849e-1c68b47ea721" path="/var/lib/kubelet/pods/5522a57b-9892-4cd8-849e-1c68b47ea721/volumes"
Nov 03 19:59:47.272976 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:47.272937    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55"]
Nov 03 19:59:47.273736 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:47.273720    5591 cpu_manager.go:414] "containerMap: removing container" podUID="5522a57b-9892-4cd8-849e-1c68b47ea721" containerName="payload-pre-processing"
Nov 03 19:59:47.273785 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:47.273738    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5522a57b-9892-4cd8-849e-1c68b47ea721" containerName="payload-pre-processing"
Nov 03 19:59:47.273785 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:47.273778    5591 cpu_manager.go:414] "containerMap: removing container" podUID="8656a4cc-537b-47c7-9429-6581fb9ab565" containerName="payload-processing"
Nov 03 19:59:47.273785 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:47.273784    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8656a4cc-537b-47c7-9429-6581fb9ab565" containerName="payload-processing"
Nov 03 19:59:47.273913 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:47.273896    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8656a4cc-537b-47c7-9429-6581fb9ab565" containerName="payload-processing"
Nov 03 19:59:47.273913 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:47.273915    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="5522a57b-9892-4cd8-849e-1c68b47ea721" containerName="payload-pre-processing"
Nov 03 19:59:47.279347 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:47.279328    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55"
Nov 03 19:59:47.292253 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:47.292228    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55"]
Nov 03 19:59:47.296547 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:47.296521    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c6092e-dockercfg-xmq8r\""
Nov 03 19:59:47.296713 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:47.296599    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c6092e\""
Nov 03 19:59:47.666570 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:47.666275    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be582c31-becf-4674-889c-b25de0634f36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55\" (UID: \"be582c31-becf-4674-889c-b25de0634f36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55"
Nov 03 19:59:47.666570 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:47.666481    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9frvl\" (UniqueName: \"kubernetes.io/projected/be582c31-becf-4674-889c-b25de0634f36-kube-api-access-9frvl\") pod \"payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55\" (UID: \"be582c31-becf-4674-889c-b25de0634f36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55"
Nov 03 19:59:47.768299 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:47.768250    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be582c31-becf-4674-889c-b25de0634f36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55\" (UID: \"be582c31-becf-4674-889c-b25de0634f36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55"
Nov 03 19:59:47.768487 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:47.768337    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9frvl\" (UniqueName: \"kubernetes.io/projected/be582c31-becf-4674-889c-b25de0634f36-kube-api-access-9frvl\") pod \"payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55\" (UID: \"be582c31-becf-4674-889c-b25de0634f36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55"
Nov 03 19:59:47.768960 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:47.768939    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be582c31-becf-4674-889c-b25de0634f36-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55\" (UID: \"be582c31-becf-4674-889c-b25de0634f36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55"
Nov 03 19:59:47.780828 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:47.780782    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9frvl\" (UniqueName: \"kubernetes.io/projected/be582c31-becf-4674-889c-b25de0634f36-kube-api-access-9frvl\") pod \"payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55\" (UID: \"be582c31-becf-4674-889c-b25de0634f36\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55"
Nov 03 19:59:47.901230 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:47.898487    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55"
Nov 03 19:59:48.415877 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:48.415850    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55"]
Nov 03 19:59:48.417271 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:59:48.417241    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe582c31_becf_4674_889c_b25de0634f36.slice/crio-07929d4ae18eb14530e3c2a7907f2c6334d87463c2ba9dfc15cd312db057510d WatchSource:0}: Error finding container 07929d4ae18eb14530e3c2a7907f2c6334d87463c2ba9dfc15cd312db057510d: Status 404 returned error can't find the container with id 07929d4ae18eb14530e3c2a7907f2c6334d87463c2ba9dfc15cd312db057510d
Nov 03 19:59:48.968866 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:48.968829    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8"]
Nov 03 19:59:48.975738 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:48.975678    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8"
Nov 03 19:59:48.980039 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:48.980014    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-5cf30a-serving-cert\""
Nov 03 19:59:48.986986 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:48.986961    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8"]
Nov 03 19:59:49.015561 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:49.015523    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8\" (UID: \"12d2b7e4-96b2-4ee9-a511-4ca0f21228fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8"
Nov 03 19:59:49.015767 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:49.015578    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8\" (UID: \"12d2b7e4-96b2-4ee9-a511-4ca0f21228fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8"
Nov 03 19:59:49.015819 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:49.015787    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hcb9\" (UniqueName: \"kubernetes.io/projected/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-kube-api-access-8hcb9\") pod \"maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8\" (UID: \"12d2b7e4-96b2-4ee9-a511-4ca0f21228fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8"
Nov 03 19:59:49.015870 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:49.015838    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-metrics-certs\") pod \"maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8\" (UID: \"12d2b7e4-96b2-4ee9-a511-4ca0f21228fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8"
Nov 03 19:59:49.125659 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:49.125062    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-metrics-certs\") pod \"maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8\" (UID: \"12d2b7e4-96b2-4ee9-a511-4ca0f21228fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8"
Nov 03 19:59:49.125659 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:49.125368    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8\" (UID: \"12d2b7e4-96b2-4ee9-a511-4ca0f21228fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8"
Nov 03 19:59:49.125659 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:49.125411    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8\" (UID: \"12d2b7e4-96b2-4ee9-a511-4ca0f21228fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8"
Nov 03 19:59:49.127012 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:49.126113    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hcb9\" (UniqueName: \"kubernetes.io/projected/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-kube-api-access-8hcb9\") pod \"maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8\" (UID: \"12d2b7e4-96b2-4ee9-a511-4ca0f21228fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8"
Nov 03 19:59:49.127352 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:59:49.127332    5591 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-serving-cert: secret "maas-api-e2e-praxis-b-5cf30a-serving-cert" not found
Nov 03 19:59:49.127555 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:59:49.127538    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-maas-api-tls podName:12d2b7e4-96b2-4ee9-a511-4ca0f21228fe nodeName:}" failed. No retries permitted until 2026-11-03 19:59:49.627514014 +0000 UTC m=+1338.399807025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-maas-api-tls") pod "maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8" (UID: "12d2b7e4-96b2-4ee9-a511-4ca0f21228fe") : secret "maas-api-e2e-praxis-b-5cf30a-serving-cert" not found
Nov 03 19:59:49.128892 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:49.128868    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8\" (UID: \"12d2b7e4-96b2-4ee9-a511-4ca0f21228fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8"
Nov 03 19:59:49.149424 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:49.149391    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-metrics-certs\") pod \"maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8\" (UID: \"12d2b7e4-96b2-4ee9-a511-4ca0f21228fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8"
Nov 03 19:59:49.150319 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:49.149939    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hcb9\" (UniqueName: \"kubernetes.io/projected/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-kube-api-access-8hcb9\") pod \"maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8\" (UID: \"12d2b7e4-96b2-4ee9-a511-4ca0f21228fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8"
Nov 03 19:59:49.216764 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:49.216530    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55" event={"ID":"be582c31-becf-4674-889c-b25de0634f36","Type":"ContainerStarted","Data":"7d4e2d119f4d92326b56ae2654dc377f889e1976efa63e98485684e0c19a2cc0"}
Nov 03 19:59:49.216764 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:49.216595    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55" event={"ID":"be582c31-becf-4674-889c-b25de0634f36","Type":"ContainerStarted","Data":"07929d4ae18eb14530e3c2a7907f2c6334d87463c2ba9dfc15cd312db057510d"}
Nov 03 19:59:49.217595 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:49.217577    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55"
Nov 03 19:59:49.238329 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:49.238198    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55" podStartSLOduration=2.238182797 podStartE2EDuration="2.238182797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:49.23565725 +0000 UTC m=+1338.007950271" watchObservedRunningTime="2026-11-03 19:59:49.238182797 +0000 UTC m=+1338.010475815"
Nov 03 19:59:49.646083 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:49.646046    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8\" (UID: \"12d2b7e4-96b2-4ee9-a511-4ca0f21228fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8"
Nov 03 19:59:49.648809 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:49.648778    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8\" (UID: \"12d2b7e4-96b2-4ee9-a511-4ca0f21228fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8"
Nov 03 19:59:49.898852 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:49.898518    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8"
Nov 03 19:59:50.234149 ip-10-0-143-206 kubenswrapper[5591]: W1103 19:59:50.234115    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12d2b7e4_96b2_4ee9_a511_4ca0f21228fe.slice/crio-329057f5c491c8b0a9b6932d69decd31ae5ae18efe34d957490b9452c96a083d WatchSource:0}: Error finding container 329057f5c491c8b0a9b6932d69decd31ae5ae18efe34d957490b9452c96a083d: Status 404 returned error can't find the container with id 329057f5c491c8b0a9b6932d69decd31ae5ae18efe34d957490b9452c96a083d
Nov 03 19:59:50.234301 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:50.234198    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8"]
Nov 03 19:59:51.077983 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:51.077893    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf"
Nov 03 19:59:51.233347 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:51.233313    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8" event={"ID":"12d2b7e4-96b2-4ee9-a511-4ca0f21228fe","Type":"ContainerStarted","Data":"c9bb286110ece406e2f59a98d9ba3c6505a2bf9ddfdf152642718d16e2f14e89"}
Nov 03 19:59:51.233535 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:51.233374    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8" event={"ID":"12d2b7e4-96b2-4ee9-a511-4ca0f21228fe","Type":"ContainerStarted","Data":"329057f5c491c8b0a9b6932d69decd31ae5ae18efe34d957490b9452c96a083d"}
Nov 03 19:59:51.233736 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:51.233714    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8"
Nov 03 19:59:51.254825 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:51.254774    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8" podStartSLOduration=2.674581298 podStartE2EDuration="3.254758649s" totalImagesPullingTime="580.177351ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:59:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:59:51.252345217 +0000 UTC m=+1340.024638236" watchObservedRunningTime="2026-11-03 19:59:51.254758649 +0000 UTC m=+1340.027051667"
Nov 03 19:59:52.087629 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:52.087600    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6"
Nov 03 19:59:53.210011 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:53.209965    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g"]
Nov 03 19:59:53.210525 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:53.210251    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g" podUID="79b1ecb5-d452-4402-a4bf-9e89e9e67131" containerName="maas-api" containerID="cri-o://8951af66b1f169247e4c762c57f0f3da58f489ec07b4f7197d2ed250240edee9" gracePeriod=30
Nov 03 19:59:53.807014 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:53.806986    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g"
Nov 03 19:59:53.937354 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:53.937311    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79b1ecb5-d452-4402-a4bf-9e89e9e67131-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b1ecb5-d452-4402-a4bf-9e89e9e67131-maas-api-tls\") pod \"79b1ecb5-d452-4402-a4bf-9e89e9e67131\" (UID: \"79b1ecb5-d452-4402-a4bf-9e89e9e67131\") "
Nov 03 19:59:53.937576 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:53.937557    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b1ecb5-d452-4402-a4bf-9e89e9e67131-kube-api-access-p6fcj\" (UniqueName: \"kubernetes.io/projected/79b1ecb5-d452-4402-a4bf-9e89e9e67131-kube-api-access-p6fcj\") pod \"79b1ecb5-d452-4402-a4bf-9e89e9e67131\" (UID: \"79b1ecb5-d452-4402-a4bf-9e89e9e67131\") "
Nov 03 19:59:53.937680 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:53.937624    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79b1ecb5-d452-4402-a4bf-9e89e9e67131-metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b1ecb5-d452-4402-a4bf-9e89e9e67131-metrics-certs\") pod \"79b1ecb5-d452-4402-a4bf-9e89e9e67131\" (UID: \"79b1ecb5-d452-4402-a4bf-9e89e9e67131\") "
Nov 03 19:59:53.937817 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:53.937770    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79b1ecb5-d452-4402-a4bf-9e89e9e67131-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b1ecb5-d452-4402-a4bf-9e89e9e67131-openshift-service-ca\") pod \"79b1ecb5-d452-4402-a4bf-9e89e9e67131\" (UID: \"79b1ecb5-d452-4402-a4bf-9e89e9e67131\") "
Nov 03 19:59:53.938403 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:53.938378    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b1ecb5-d452-4402-a4bf-9e89e9e67131-openshift-service-ca" pod "79b1ecb5-d452-4402-a4bf-9e89e9e67131" (UID: "79b1ecb5-d452-4402-a4bf-9e89e9e67131"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:59:53.944254 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:53.942783    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b1ecb5-d452-4402-a4bf-9e89e9e67131-metrics-certs" pod "79b1ecb5-d452-4402-a4bf-9e89e9e67131" (UID: "79b1ecb5-d452-4402-a4bf-9e89e9e67131"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:59:53.944254 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:53.942802    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b1ecb5-d452-4402-a4bf-9e89e9e67131-kube-api-access-p6fcj" pod "79b1ecb5-d452-4402-a4bf-9e89e9e67131" (UID: "79b1ecb5-d452-4402-a4bf-9e89e9e67131"). InnerVolumeSpecName "kube-api-access-p6fcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:59:53.944254 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:53.944071    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b1ecb5-d452-4402-a4bf-9e89e9e67131-maas-api-tls" pod "79b1ecb5-d452-4402-a4bf-9e89e9e67131" (UID: "79b1ecb5-d452-4402-a4bf-9e89e9e67131"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:59:54.040541 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:54.040458    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6fcj\" (UniqueName: \"kubernetes.io/projected/79b1ecb5-d452-4402-a4bf-9e89e9e67131-kube-api-access-p6fcj\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:54.040541 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:54.040491    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b1ecb5-d452-4402-a4bf-9e89e9e67131-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:54.040541 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:54.040501    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b1ecb5-d452-4402-a4bf-9e89e9e67131-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:54.040541 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:54.040510    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b1ecb5-d452-4402-a4bf-9e89e9e67131-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 19:59:54.320836 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:54.317721    5591 generic.go:356] "Generic (PLEG): container finished" podID="79b1ecb5-d452-4402-a4bf-9e89e9e67131" containerID="8951af66b1f169247e4c762c57f0f3da58f489ec07b4f7197d2ed250240edee9" exitCode=0
Nov 03 19:59:54.320836 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:54.317830    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g" event={"ID":"79b1ecb5-d452-4402-a4bf-9e89e9e67131","Type":"ContainerDied","Data":"8951af66b1f169247e4c762c57f0f3da58f489ec07b4f7197d2ed250240edee9"}
Nov 03 19:59:54.320836 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:54.317932    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g" event={"ID":"79b1ecb5-d452-4402-a4bf-9e89e9e67131","Type":"ContainerDied","Data":"9765c2e8b7506824f681f3babb1a506c7b470a9beb76bde20fcee413b83ae83a"}
Nov 03 19:59:54.320836 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:54.317968    5591 scope.go:122] "RemoveContainer" containerID="8951af66b1f169247e4c762c57f0f3da58f489ec07b4f7197d2ed250240edee9"
Nov 03 19:59:54.320836 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:54.318189    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g"
Nov 03 19:59:54.343746 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:54.343723    5591 scope.go:122] "RemoveContainer" containerID="8951af66b1f169247e4c762c57f0f3da58f489ec07b4f7197d2ed250240edee9"
Nov 03 19:59:54.344084 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:59:54.344056    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8951af66b1f169247e4c762c57f0f3da58f489ec07b4f7197d2ed250240edee9\": container with ID starting with 8951af66b1f169247e4c762c57f0f3da58f489ec07b4f7197d2ed250240edee9 not found: ID does not exist" containerID="8951af66b1f169247e4c762c57f0f3da58f489ec07b4f7197d2ed250240edee9"
Nov 03 19:59:54.344189 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:54.344098    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8951af66b1f169247e4c762c57f0f3da58f489ec07b4f7197d2ed250240edee9"} err="failed to get container status \"8951af66b1f169247e4c762c57f0f3da58f489ec07b4f7197d2ed250240edee9\": rpc error: code = NotFound desc = could not find container \"8951af66b1f169247e4c762c57f0f3da58f489ec07b4f7197d2ed250240edee9\": container with ID starting with 8951af66b1f169247e4c762c57f0f3da58f489ec07b4f7197d2ed250240edee9 not found: ID does not exist"
Nov 03 19:59:54.354617 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:54.354596    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g"]
Nov 03 19:59:54.358462 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:54.358441    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b6e2b-679c98ccf-v884g"]
Nov 03 19:59:55.138387 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:55.138352    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"]
Nov 03 19:59:55.138599 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:55.138562    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq" podUID="f226a9f7-2809-4890-8cc3-29013e49d774" containerName="istio-proxy" containerID="cri-o://6a575b26f292570d4f7d08bcbac3c47c245a49ce6835d60ed76f5bf50e1297e5" gracePeriod=30
Nov 03 19:59:55.397282 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:55.397110    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79b1ecb5-d452-4402-a4bf-9e89e9e67131" path="/var/lib/kubelet/pods/79b1ecb5-d452-4402-a4bf-9e89e9e67131/volumes"
Nov 03 19:59:57.256346 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:57.256320    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8"
Nov 03 19:59:58.726627 ip-10-0-143-206 kubenswrapper[5591]: E1103 19:59:58.725899    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 19:59:59.070114 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:59.070074    5591 patch_prober.go:29] interesting pod/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:59:59.070308 ip-10-0-143-206 kubenswrapper[5591]: I1103 19:59:59.070154    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq" podUID="f226a9f7-2809-4890-8cc3-29013e49d774" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:00:00.421602 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.421525    5591 generic.go:356] "Generic (PLEG): container finished" podID="f226a9f7-2809-4890-8cc3-29013e49d774" containerID="6a575b26f292570d4f7d08bcbac3c47c245a49ce6835d60ed76f5bf50e1297e5" exitCode=0
Nov 03 20:00:00.421998 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.421601    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq" event={"ID":"f226a9f7-2809-4890-8cc3-29013e49d774","Type":"ContainerDied","Data":"6a575b26f292570d4f7d08bcbac3c47c245a49ce6835d60ed76f5bf50e1297e5"}
Nov 03 20:00:00.421998 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.421658    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq" event={"ID":"f226a9f7-2809-4890-8cc3-29013e49d774","Type":"ContainerDied","Data":"27ca9f5c4895cfcf55aed14435bbcb9763950cb21ff1e5cbb2d52536e0cf7885"}
Nov 03 20:00:00.421998 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.421677    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27ca9f5c4895cfcf55aed14435bbcb9763950cb21ff1e5cbb2d52536e0cf7885"
Nov 03 20:00:00.421998 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.421692    5591 scope.go:122] "RemoveContainer" containerID="8842dc2ff9cb2bc31ba58bbcedbd7f0d914cd991a91fc3b5c2352b06cdb48090"
Nov 03 20:00:00.434876 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.434857    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 20:00:00.523234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.522784    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-workload-certs\") pod \"f226a9f7-2809-4890-8cc3-29013e49d774\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") "
Nov 03 20:00:00.523234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.522843    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f226a9f7-2809-4890-8cc3-29013e49d774-kube-api-access-5zjcw\" (UniqueName: \"kubernetes.io/projected/f226a9f7-2809-4890-8cc3-29013e49d774-kube-api-access-5zjcw\") pod \"f226a9f7-2809-4890-8cc3-29013e49d774\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") "
Nov 03 20:00:00.523234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.522873    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-workload-socket\") pod \"f226a9f7-2809-4890-8cc3-29013e49d774\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") "
Nov 03 20:00:00.523234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.522912    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-istio-data\") pod \"f226a9f7-2809-4890-8cc3-29013e49d774\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") "
Nov 03 20:00:00.523234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.522987    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-credential-socket\") pod \"f226a9f7-2809-4890-8cc3-29013e49d774\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") "
Nov 03 20:00:00.523234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.523046    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f226a9f7-2809-4890-8cc3-29013e49d774-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f226a9f7-2809-4890-8cc3-29013e49d774-istio-podinfo\") pod \"f226a9f7-2809-4890-8cc3-29013e49d774\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") "
Nov 03 20:00:00.523234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.523131    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f226a9f7-2809-4890-8cc3-29013e49d774-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f226a9f7-2809-4890-8cc3-29013e49d774-istiod-ca-cert\") pod \"f226a9f7-2809-4890-8cc3-29013e49d774\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") "
Nov 03 20:00:00.523234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.523181    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-istio-envoy\") pod \"f226a9f7-2809-4890-8cc3-29013e49d774\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") "
Nov 03 20:00:00.523767 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.523404    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f226a9f7-2809-4890-8cc3-29013e49d774-istio-token\" (UniqueName: \"kubernetes.io/projected/f226a9f7-2809-4890-8cc3-29013e49d774-istio-token\") pod \"f226a9f7-2809-4890-8cc3-29013e49d774\" (UID: \"f226a9f7-2809-4890-8cc3-29013e49d774\") "
Nov 03 20:00:00.523767 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.523499    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-istio-data" pod "f226a9f7-2809-4890-8cc3-29013e49d774" (UID: "f226a9f7-2809-4890-8cc3-29013e49d774"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:00:00.523767 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.523696    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-workload-certs" pod "f226a9f7-2809-4890-8cc3-29013e49d774" (UID: "f226a9f7-2809-4890-8cc3-29013e49d774"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:00:00.534271 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.524244    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:00.534271 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.524269    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:00.534271 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.526413    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f226a9f7-2809-4890-8cc3-29013e49d774-istiod-ca-cert" pod "f226a9f7-2809-4890-8cc3-29013e49d774" (UID: "f226a9f7-2809-4890-8cc3-29013e49d774"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:00.543239 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.541674    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-istio-envoy" pod "f226a9f7-2809-4890-8cc3-29013e49d774" (UID: "f226a9f7-2809-4890-8cc3-29013e49d774"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:00:00.553119 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.552297    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-credential-socket" pod "f226a9f7-2809-4890-8cc3-29013e49d774" (UID: "f226a9f7-2809-4890-8cc3-29013e49d774"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:00:00.553119 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.552392    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-workload-socket" pod "f226a9f7-2809-4890-8cc3-29013e49d774" (UID: "f226a9f7-2809-4890-8cc3-29013e49d774"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:00:00.563730 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.562675    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f226a9f7-2809-4890-8cc3-29013e49d774-istio-token" pod "f226a9f7-2809-4890-8cc3-29013e49d774" (UID: "f226a9f7-2809-4890-8cc3-29013e49d774"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:00.563730 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.562814    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f226a9f7-2809-4890-8cc3-29013e49d774-kube-api-access-5zjcw" pod "f226a9f7-2809-4890-8cc3-29013e49d774" (UID: "f226a9f7-2809-4890-8cc3-29013e49d774"). InnerVolumeSpecName "kube-api-access-5zjcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:00.565672 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.565642    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f226a9f7-2809-4890-8cc3-29013e49d774-istio-podinfo" pod "f226a9f7-2809-4890-8cc3-29013e49d774" (UID: "f226a9f7-2809-4890-8cc3-29013e49d774"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:00:00.626249 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.626216    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f226a9f7-2809-4890-8cc3-29013e49d774-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:00.626249 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.626248    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zjcw\" (UniqueName: \"kubernetes.io/projected/f226a9f7-2809-4890-8cc3-29013e49d774-kube-api-access-5zjcw\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:00.626469 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.626260    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:00.626469 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.626269    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:00.626469 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.626277    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f226a9f7-2809-4890-8cc3-29013e49d774-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:00.626469 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.626285    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f226a9f7-2809-4890-8cc3-29013e49d774-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:00.626469 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:00.626299    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f226a9f7-2809-4890-8cc3-29013e49d774-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:01.236631 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:01.236605    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55"
Nov 03 20:00:01.430719 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:01.430689    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"
Nov 03 20:00:01.566415 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:01.566374    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"]
Nov 03 20:00:01.577842 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:01.575441    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b6e2b-openshift-default-b66f9754c-2d4jq"]
Nov 03 20:00:03.023500 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.023462    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"]
Nov 03 20:00:03.024796 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.024748    5591 cpu_manager.go:414] "containerMap: removing container" podUID="f226a9f7-2809-4890-8cc3-29013e49d774" containerName="istio-proxy"
Nov 03 20:00:03.025042 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.025002    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f226a9f7-2809-4890-8cc3-29013e49d774" containerName="istio-proxy"
Nov 03 20:00:03.025283 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.025251    5591 cpu_manager.go:414] "containerMap: removing container" podUID="79b1ecb5-d452-4402-a4bf-9e89e9e67131" containerName="maas-api"
Nov 03 20:00:03.025436 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.025404    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b1ecb5-d452-4402-a4bf-9e89e9e67131" containerName="maas-api"
Nov 03 20:00:03.025822 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.025803    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f226a9f7-2809-4890-8cc3-29013e49d774" containerName="istio-proxy"
Nov 03 20:00:03.025953 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.025941    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f226a9f7-2809-4890-8cc3-29013e49d774" containerName="istio-proxy"
Nov 03 20:00:03.026075 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.026056    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b1ecb5-d452-4402-a4bf-9e89e9e67131" containerName="maas-api"
Nov 03 20:00:03.032143 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.032119    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.041071 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.041041    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"]
Nov 03 20:00:03.043504 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.043465    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8b3020-openshift-default-dockercfg-8fhsl\""
Nov 03 20:00:03.137540 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.134113    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnhhn\" (UniqueName: \"kubernetes.io/projected/56528ce3-95f8-408d-96db-04ac7bcab926-kube-api-access-vnhhn\") pod \"e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") " pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.137540 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.134275    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56528ce3-95f8-408d-96db-04ac7bcab926-istiod-ca-cert\") pod \"e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") " pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.137540 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.134334    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-workload-certs\") pod \"e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") " pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.137540 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.134376    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-workload-socket\") pod \"e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") " pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.137540 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.134459    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56528ce3-95f8-408d-96db-04ac7bcab926-istio-token\") pod \"e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") " pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.137540 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.134584    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56528ce3-95f8-408d-96db-04ac7bcab926-istio-podinfo\") pod \"e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") " pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.137540 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.134612    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-istio-data\") pod \"e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") " pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.137540 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.134675    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-credential-socket\") pod \"e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") " pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.137540 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.134779    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-istio-envoy\") pod \"e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") " pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.240230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.239245    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56528ce3-95f8-408d-96db-04ac7bcab926-istiod-ca-cert\") pod \"e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") " pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.240230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.239313    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-workload-certs\") pod \"e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") " pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.240230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.239354    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-workload-socket\") pod \"e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") " pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.240230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.239424    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56528ce3-95f8-408d-96db-04ac7bcab926-istio-token\") pod \"e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") " pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.240230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.239520    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56528ce3-95f8-408d-96db-04ac7bcab926-istio-podinfo\") pod \"e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") " pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.240230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.239551    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-istio-data\") pod \"e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") " pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.240230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.239595    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-credential-socket\") pod \"e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") " pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.240230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.239682    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-istio-envoy\") pod \"e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") " pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.240230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.239842    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnhhn\" (UniqueName: \"kubernetes.io/projected/56528ce3-95f8-408d-96db-04ac7bcab926-kube-api-access-vnhhn\") pod \"e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") " pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.244226 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.241001    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56528ce3-95f8-408d-96db-04ac7bcab926-istiod-ca-cert\") pod \"e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") " pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.244226 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.241308    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-workload-certs\") pod \"e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") " pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.244226 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.241543    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-workload-socket\") pod \"e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") " pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.248226 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.247532    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-credential-socket\") pod \"e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") " pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.248226 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.247745    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-istio-data\") pod \"e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") " pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.252231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.250691    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-istio-envoy\") pod \"e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") " pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.256236 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.255789    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnhhn\" (UniqueName: \"kubernetes.io/projected/56528ce3-95f8-408d-96db-04ac7bcab926-kube-api-access-vnhhn\") pod \"e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") " pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.256236 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.255806    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56528ce3-95f8-408d-96db-04ac7bcab926-istio-token\") pod \"e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") " pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.280275 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.275145    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56528ce3-95f8-408d-96db-04ac7bcab926-istio-podinfo\") pod \"e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") " pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.379001 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.375684    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:03.413248 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.408669    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f226a9f7-2809-4890-8cc3-29013e49d774" path="/var/lib/kubelet/pods/f226a9f7-2809-4890-8cc3-29013e49d774/volumes"
Nov 03 20:00:03.718029 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.717997    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"]
Nov 03 20:00:03.719562 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:00:03.719536    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56528ce3_95f8_408d_96db_04ac7bcab926.slice/crio-dd1baebcc030cf98d87d15fdb8a78557c99cec4e26dc6094923669907bfbe9e9 WatchSource:0}: Error finding container dd1baebcc030cf98d87d15fdb8a78557c99cec4e26dc6094923669907bfbe9e9: Status 404 returned error can't find the container with id dd1baebcc030cf98d87d15fdb8a78557c99cec4e26dc6094923669907bfbe9e9
Nov 03 20:00:03.722847 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.722807    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:00:03.722923 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.722895    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:00:03.722973 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:03.722938    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:00:04.507288 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:04.506906    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x" event={"ID":"56528ce3-95f8-408d-96db-04ac7bcab926","Type":"ContainerStarted","Data":"bc51a650762e67653343b8311b56e1151c2fbe39cb177e5f812fc9753ed3be0e"}
Nov 03 20:00:04.507288 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:04.506974    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x" event={"ID":"56528ce3-95f8-408d-96db-04ac7bcab926","Type":"ContainerStarted","Data":"dd1baebcc030cf98d87d15fdb8a78557c99cec4e26dc6094923669907bfbe9e9"}
Nov 03 20:00:04.542022 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:04.541975    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x" podStartSLOduration=2.541957617 podStartE2EDuration="2.541957617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:04.540974827 +0000 UTC m=+1353.313267846" watchObservedRunningTime="2026-11-03 20:00:04.541957617 +0000 UTC m=+1353.314250637"
Nov 03 20:00:04.944321 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:04.944281    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:00:05.376743 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:05.376713    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:05.382703 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:05.382677    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:05.545370 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:05.543113    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:05.545370 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:05.545193    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:09.377939 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:09.377912    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz"]
Nov 03 20:00:09.378732 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:09.378716    5591 cpu_manager.go:414] "containerMap: removing container" podUID="f226a9f7-2809-4890-8cc3-29013e49d774" containerName="istio-proxy"
Nov 03 20:00:09.378791 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:09.378734    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f226a9f7-2809-4890-8cc3-29013e49d774" containerName="istio-proxy"
Nov 03 20:00:09.383554 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:09.383536    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz"
Nov 03 20:00:09.387401 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:09.387374    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8b3020\""
Nov 03 20:00:09.391968 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:09.391930    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz"]
Nov 03 20:00:09.460246 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:09.454811    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z7xm\" (UniqueName: \"kubernetes.io/projected/e1591b82-68d7-4343-980b-8d6e1cce458c-kube-api-access-7z7xm\") pod \"payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz\" (UID: \"e1591b82-68d7-4343-980b-8d6e1cce458c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz"
Nov 03 20:00:09.460246 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:09.454948    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1591b82-68d7-4343-980b-8d6e1cce458c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz\" (UID: \"e1591b82-68d7-4343-980b-8d6e1cce458c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz"
Nov 03 20:00:09.469326 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:09.463550    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8b3020-dockercfg-7ghgb\""
Nov 03 20:00:09.563231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:09.557425    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z7xm\" (UniqueName: \"kubernetes.io/projected/e1591b82-68d7-4343-980b-8d6e1cce458c-kube-api-access-7z7xm\") pod \"payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz\" (UID: \"e1591b82-68d7-4343-980b-8d6e1cce458c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz"
Nov 03 20:00:09.563231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:09.557531    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1591b82-68d7-4343-980b-8d6e1cce458c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz\" (UID: \"e1591b82-68d7-4343-980b-8d6e1cce458c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz"
Nov 03 20:00:09.563231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:09.558562    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1591b82-68d7-4343-980b-8d6e1cce458c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz\" (UID: \"e1591b82-68d7-4343-980b-8d6e1cce458c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz"
Nov 03 20:00:09.676279 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:09.675541    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z7xm\" (UniqueName: \"kubernetes.io/projected/e1591b82-68d7-4343-980b-8d6e1cce458c-kube-api-access-7z7xm\") pod \"payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz\" (UID: \"e1591b82-68d7-4343-980b-8d6e1cce458c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz"
Nov 03 20:00:09.798431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:09.757951    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2"]
Nov 03 20:00:09.798431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:09.783077    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2"
Nov 03 20:00:09.798431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:09.794753    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2"]
Nov 03 20:00:09.862304 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:09.842855    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz"
Nov 03 20:00:09.889534 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:09.887542    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h2qf\" (UniqueName: \"kubernetes.io/projected/ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f-kube-api-access-6h2qf\") pod \"payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2\" (UID: \"ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2"
Nov 03 20:00:09.889534 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:09.887690    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2\" (UID: \"ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2"
Nov 03 20:00:09.990535 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:09.989734    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h2qf\" (UniqueName: \"kubernetes.io/projected/ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f-kube-api-access-6h2qf\") pod \"payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2\" (UID: \"ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2"
Nov 03 20:00:09.990535 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:09.989857    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2\" (UID: \"ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2"
Nov 03 20:00:09.991046 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:09.991024    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2\" (UID: \"ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2"
Nov 03 20:00:10.002874 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:10.002842    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h2qf\" (UniqueName: \"kubernetes.io/projected/ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f-kube-api-access-6h2qf\") pod \"payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2\" (UID: \"ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2"
Nov 03 20:00:10.156083 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:10.156053    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2"
Nov 03 20:00:10.163147 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:00:10.163117    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1591b82_68d7_4343_980b_8d6e1cce458c.slice/crio-320d7955fc0bbe8e2756b9280300fe189a32696606d07cb9c4c345f1902aad7a WatchSource:0}: Error finding container 320d7955fc0bbe8e2756b9280300fe189a32696606d07cb9c4c345f1902aad7a: Status 404 returned error can't find the container with id 320d7955fc0bbe8e2756b9280300fe189a32696606d07cb9c4c345f1902aad7a
Nov 03 20:00:10.165555 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:10.165532    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz"]
Nov 03 20:00:10.459855 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:10.459827    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2"]
Nov 03 20:00:10.460698 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:00:10.460672    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb21a4b_a11b_4ff9_89d5_2e2edc982b9f.slice/crio-c14daf634678d1fb872561ccd7dc8f4061c2850d4c35c47eb0c065cffed70d95 WatchSource:0}: Error finding container c14daf634678d1fb872561ccd7dc8f4061c2850d4c35c47eb0c065cffed70d95: Status 404 returned error can't find the container with id c14daf634678d1fb872561ccd7dc8f4061c2850d4c35c47eb0c065cffed70d95
Nov 03 20:00:10.714292 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:10.710872    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz" event={"ID":"e1591b82-68d7-4343-980b-8d6e1cce458c","Type":"ContainerStarted","Data":"be9d8e0733db08b11a3b85739ef791c1d9036f3949518948e7492d1185d913f4"}
Nov 03 20:00:10.714292 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:10.710940    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz" event={"ID":"e1591b82-68d7-4343-980b-8d6e1cce458c","Type":"ContainerStarted","Data":"320d7955fc0bbe8e2756b9280300fe189a32696606d07cb9c4c345f1902aad7a"}
Nov 03 20:00:10.714292 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:10.712234    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz"
Nov 03 20:00:10.716540 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:10.715853    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2" event={"ID":"ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f","Type":"ContainerStarted","Data":"ebf7e4e956280a36943b58f09aea51c60c94cdf0538e2e590ec0f6b87fc453f8"}
Nov 03 20:00:10.716540 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:10.715911    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2" event={"ID":"ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f","Type":"ContainerStarted","Data":"c14daf634678d1fb872561ccd7dc8f4061c2850d4c35c47eb0c065cffed70d95"}
Nov 03 20:00:10.716874 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:10.716844    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2"
Nov 03 20:00:10.752315 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:10.751233    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz" podStartSLOduration=1.7511896390000001 podStartE2EDuration="1.751189639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:10.736980389 +0000 UTC m=+1359.509273430" watchObservedRunningTime="2026-11-03 20:00:10.751189639 +0000 UTC m=+1359.523482658"
Nov 03 20:00:10.768298 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:10.766645    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2" podStartSLOduration=1.7666258350000001 podStartE2EDuration="1.766625835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:10.756800481 +0000 UTC m=+1359.529093501" watchObservedRunningTime="2026-11-03 20:00:10.766625835 +0000 UTC m=+1359.538918855"
Nov 03 20:00:13.864027 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:13.863943    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r"]
Nov 03 20:00:13.864613 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:13.864303    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r" podUID="48822eec-0744-4d8b-983a-9129d93987fc" containerName="maas-api" containerID="cri-o://190c3df423a3809f4a58b98bfcc54d2284c7ac3b11199d403c6b7eb06bd55f12" gracePeriod=30
Nov 03 20:00:14.434251 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:14.425820    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r"
Nov 03 20:00:14.584375 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:14.584287    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48822eec-0744-4d8b-983a-9129d93987fc-kube-api-access-6889d\" (UniqueName: \"kubernetes.io/projected/48822eec-0744-4d8b-983a-9129d93987fc-kube-api-access-6889d\") pod \"48822eec-0744-4d8b-983a-9129d93987fc\" (UID: \"48822eec-0744-4d8b-983a-9129d93987fc\") "
Nov 03 20:00:14.584573 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:14.584554    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48822eec-0744-4d8b-983a-9129d93987fc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48822eec-0744-4d8b-983a-9129d93987fc-openshift-service-ca\") pod \"48822eec-0744-4d8b-983a-9129d93987fc\" (UID: \"48822eec-0744-4d8b-983a-9129d93987fc\") "
Nov 03 20:00:14.584651 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:14.584612    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48822eec-0744-4d8b-983a-9129d93987fc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/48822eec-0744-4d8b-983a-9129d93987fc-metrics-certs\") pod \"48822eec-0744-4d8b-983a-9129d93987fc\" (UID: \"48822eec-0744-4d8b-983a-9129d93987fc\") "
Nov 03 20:00:14.584741 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:14.584713    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48822eec-0744-4d8b-983a-9129d93987fc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48822eec-0744-4d8b-983a-9129d93987fc-maas-api-tls\") pod \"48822eec-0744-4d8b-983a-9129d93987fc\" (UID: \"48822eec-0744-4d8b-983a-9129d93987fc\") "
Nov 03 20:00:14.585006 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:14.584973    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48822eec-0744-4d8b-983a-9129d93987fc-openshift-service-ca" pod "48822eec-0744-4d8b-983a-9129d93987fc" (UID: "48822eec-0744-4d8b-983a-9129d93987fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:14.585357 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:14.585335    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48822eec-0744-4d8b-983a-9129d93987fc-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:14.587327 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:14.587306    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48822eec-0744-4d8b-983a-9129d93987fc-maas-api-tls" pod "48822eec-0744-4d8b-983a-9129d93987fc" (UID: "48822eec-0744-4d8b-983a-9129d93987fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:00:14.587430 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:14.587415    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48822eec-0744-4d8b-983a-9129d93987fc-metrics-certs" pod "48822eec-0744-4d8b-983a-9129d93987fc" (UID: "48822eec-0744-4d8b-983a-9129d93987fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:00:14.587498 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:14.587476    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48822eec-0744-4d8b-983a-9129d93987fc-kube-api-access-6889d" pod "48822eec-0744-4d8b-983a-9129d93987fc" (UID: "48822eec-0744-4d8b-983a-9129d93987fc"). InnerVolumeSpecName "kube-api-access-6889d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:14.689231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:14.688383    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48822eec-0744-4d8b-983a-9129d93987fc-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:14.689231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:14.688417    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48822eec-0744-4d8b-983a-9129d93987fc-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:14.689231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:14.688433    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6889d\" (UniqueName: \"kubernetes.io/projected/48822eec-0744-4d8b-983a-9129d93987fc-kube-api-access-6889d\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:14.787258 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:14.781831    5591 generic.go:356] "Generic (PLEG): container finished" podID="48822eec-0744-4d8b-983a-9129d93987fc" containerID="190c3df423a3809f4a58b98bfcc54d2284c7ac3b11199d403c6b7eb06bd55f12" exitCode=0
Nov 03 20:00:14.787258 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:14.781937    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r" event={"ID":"48822eec-0744-4d8b-983a-9129d93987fc","Type":"ContainerDied","Data":"190c3df423a3809f4a58b98bfcc54d2284c7ac3b11199d403c6b7eb06bd55f12"}
Nov 03 20:00:14.787258 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:14.781987    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r" event={"ID":"48822eec-0744-4d8b-983a-9129d93987fc","Type":"ContainerDied","Data":"4a8bc811817f99068bb8c09f1997e199c6dcd9e3ccb21f2e6c2eb22c4683fc72"}
Nov 03 20:00:14.787258 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:14.782019    5591 scope.go:122] "RemoveContainer" containerID="190c3df423a3809f4a58b98bfcc54d2284c7ac3b11199d403c6b7eb06bd55f12"
Nov 03 20:00:14.787258 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:14.782189    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r"
Nov 03 20:00:14.929413 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:14.929377    5591 scope.go:122] "RemoveContainer" containerID="190c3df423a3809f4a58b98bfcc54d2284c7ac3b11199d403c6b7eb06bd55f12"
Nov 03 20:00:14.932403 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:14.929953    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r"]
Nov 03 20:00:14.975227 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:00:14.973198    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"190c3df423a3809f4a58b98bfcc54d2284c7ac3b11199d403c6b7eb06bd55f12\": container with ID starting with 190c3df423a3809f4a58b98bfcc54d2284c7ac3b11199d403c6b7eb06bd55f12 not found: ID does not exist" containerID="190c3df423a3809f4a58b98bfcc54d2284c7ac3b11199d403c6b7eb06bd55f12"
Nov 03 20:00:14.975227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:14.973283    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"190c3df423a3809f4a58b98bfcc54d2284c7ac3b11199d403c6b7eb06bd55f12"} err="failed to get container status \"190c3df423a3809f4a58b98bfcc54d2284c7ac3b11199d403c6b7eb06bd55f12\": rpc error: code = NotFound desc = could not find container \"190c3df423a3809f4a58b98bfcc54d2284c7ac3b11199d403c6b7eb06bd55f12\": container with ID starting with 190c3df423a3809f4a58b98bfcc54d2284c7ac3b11199d403c6b7eb06bd55f12 not found: ID does not exist"
Nov 03 20:00:14.989403 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:14.986428    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-277dc9-5b67fd5c68-xqh6r"]
Nov 03 20:00:15.351745 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:15.351711    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48822eec-0744-4d8b-983a-9129d93987fc" path="/var/lib/kubelet/pods/48822eec-0744-4d8b-983a-9129d93987fc/volumes"
Nov 03 20:00:18.200528 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:18.200491    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj"]
Nov 03 20:00:18.201337 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:18.201321    5591 cpu_manager.go:414] "containerMap: removing container" podUID="48822eec-0744-4d8b-983a-9129d93987fc" containerName="maas-api"
Nov 03 20:00:18.201381 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:18.201338    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48822eec-0744-4d8b-983a-9129d93987fc" containerName="maas-api"
Nov 03 20:00:18.201486 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:18.201476    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="48822eec-0744-4d8b-983a-9129d93987fc" containerName="maas-api"
Nov 03 20:00:18.206378 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:18.206355    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj"
Nov 03 20:00:18.216372 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:18.216346    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-8b3020-serving-cert\""
Nov 03 20:00:18.220282 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:18.220259    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj"]
Nov 03 20:00:18.278063 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:18.278029    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj\" (UID: \"d141244a-fd87-4e3c-90aa-f4b1cf60aa84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj"
Nov 03 20:00:18.278351 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:18.278310    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdtlp\" (UniqueName: \"kubernetes.io/projected/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-kube-api-access-kdtlp\") pod \"maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj\" (UID: \"d141244a-fd87-4e3c-90aa-f4b1cf60aa84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj"
Nov 03 20:00:18.278416 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:18.278406    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-metrics-certs\") pod \"maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj\" (UID: \"d141244a-fd87-4e3c-90aa-f4b1cf60aa84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj"
Nov 03 20:00:18.278453 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:18.278446    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj\" (UID: \"d141244a-fd87-4e3c-90aa-f4b1cf60aa84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj"
Nov 03 20:00:18.391593 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:18.391484    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdtlp\" (UniqueName: \"kubernetes.io/projected/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-kube-api-access-kdtlp\") pod \"maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj\" (UID: \"d141244a-fd87-4e3c-90aa-f4b1cf60aa84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj"
Nov 03 20:00:18.404471 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:18.404427    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-metrics-certs\") pod \"maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj\" (UID: \"d141244a-fd87-4e3c-90aa-f4b1cf60aa84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj"
Nov 03 20:00:18.404875 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:18.404835    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj\" (UID: \"d141244a-fd87-4e3c-90aa-f4b1cf60aa84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj"
Nov 03 20:00:18.405352 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:18.405327    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj\" (UID: \"d141244a-fd87-4e3c-90aa-f4b1cf60aa84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj"
Nov 03 20:00:18.407172 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:18.407150    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj\" (UID: \"d141244a-fd87-4e3c-90aa-f4b1cf60aa84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj"
Nov 03 20:00:18.416253 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:00:18.415645    5591 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-serving-cert: secret "maas-api-e2e-ipp-b-8b3020-serving-cert" not found
Nov 03 20:00:18.416253 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:00:18.415733    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-maas-api-tls podName:d141244a-fd87-4e3c-90aa-f4b1cf60aa84 nodeName:}" failed. No retries permitted until 2026-11-03 20:00:18.915710696 +0000 UTC m=+1367.688003707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-maas-api-tls") pod "maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj" (UID: "d141244a-fd87-4e3c-90aa-f4b1cf60aa84") : secret "maas-api-e2e-ipp-b-8b3020-serving-cert" not found
Nov 03 20:00:18.440688 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:18.435093    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-metrics-certs\") pod \"maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj\" (UID: \"d141244a-fd87-4e3c-90aa-f4b1cf60aa84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj"
Nov 03 20:00:18.474775 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:18.455464    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdtlp\" (UniqueName: \"kubernetes.io/projected/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-kube-api-access-kdtlp\") pod \"maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj\" (UID: \"d141244a-fd87-4e3c-90aa-f4b1cf60aa84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj"
Nov 03 20:00:18.927712 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:18.927675    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj\" (UID: \"d141244a-fd87-4e3c-90aa-f4b1cf60aa84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj"
Nov 03 20:00:18.930445 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:18.930412    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj\" (UID: \"d141244a-fd87-4e3c-90aa-f4b1cf60aa84\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj"
Nov 03 20:00:19.146511 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:19.146478    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj"
Nov 03 20:00:19.329862 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:19.329812    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68"]
Nov 03 20:00:19.359080 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:19.358860    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m"]
Nov 03 20:00:19.542275 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:19.535075    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj"]
Nov 03 20:00:19.542275 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:00:19.536869    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd141244a_fd87_4e3c_90aa_f4b1cf60aa84.slice/crio-9a293e04ab614c6875184991343586b81bea05691af47d56bcb62ef3dbe14a22 WatchSource:0}: Error finding container 9a293e04ab614c6875184991343586b81bea05691af47d56bcb62ef3dbe14a22: Status 404 returned error can't find the container with id 9a293e04ab614c6875184991343586b81bea05691af47d56bcb62ef3dbe14a22
Nov 03 20:00:19.930284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:19.925169    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj" event={"ID":"d141244a-fd87-4e3c-90aa-f4b1cf60aa84","Type":"ContainerStarted","Data":"9a293e04ab614c6875184991343586b81bea05691af47d56bcb62ef3dbe14a22"}
Nov 03 20:00:20.938575 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:20.938536    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj" event={"ID":"d141244a-fd87-4e3c-90aa-f4b1cf60aa84","Type":"ContainerStarted","Data":"0f160857af46695d84c272391556ffec300a7d1bb9b5a8068b50493645d2615f"}
Nov 03 20:00:20.938976 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:20.938608    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj"
Nov 03 20:00:20.967761 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:20.967713    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj" podStartSLOduration=2.401788323 podStartE2EDuration="2.967694802s" totalImagesPullingTime="565.906479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:00:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:00:20.964145131 +0000 UTC m=+1369.736438149" watchObservedRunningTime="2026-11-03 20:00:20.967694802 +0000 UTC m=+1369.739987819"
Nov 03 20:00:22.831231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:22.814410    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2"
Nov 03 20:00:22.931193 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:22.894599    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz"
Nov 03 20:00:24.331466 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:24.331417    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68" podUID="4dc9dccb-784a-4082-9a7f-39223a74ce3b" containerName="payload-processing" containerID="cri-o://935f93555316f512cb6ccfa004ba4572b8a453e30499e40c1d563f3cb977f2f0" gracePeriod=25
Nov 03 20:00:24.359527 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:24.359491    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m" podUID="270abfe6-e1dc-4a2e-a106-4cd071d79612" containerName="payload-pre-processing" containerID="cri-o://d8d4f00d80fb6ad4ac1f035cb304f827106df005ad0770a87a27b6d279b16fdc" gracePeriod=25
Nov 03 20:00:24.705417 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:24.705394    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68"
Nov 03 20:00:24.709521 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:24.709499    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m"
Nov 03 20:00:24.818842 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:24.818807    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/270abfe6-e1dc-4a2e-a106-4cd071d79612-kube-api-access-dhtqw\" (UniqueName: \"kubernetes.io/projected/270abfe6-e1dc-4a2e-a106-4cd071d79612-kube-api-access-dhtqw\") pod \"270abfe6-e1dc-4a2e-a106-4cd071d79612\" (UID: \"270abfe6-e1dc-4a2e-a106-4cd071d79612\") "
Nov 03 20:00:24.819054 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:24.818920    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/270abfe6-e1dc-4a2e-a106-4cd071d79612-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/270abfe6-e1dc-4a2e-a106-4cd071d79612-plugins-config-volume\") pod \"270abfe6-e1dc-4a2e-a106-4cd071d79612\" (UID: \"270abfe6-e1dc-4a2e-a106-4cd071d79612\") "
Nov 03 20:00:24.819054 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:24.818982    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dc9dccb-784a-4082-9a7f-39223a74ce3b-kube-api-access-74cbn\" (UniqueName: \"kubernetes.io/projected/4dc9dccb-784a-4082-9a7f-39223a74ce3b-kube-api-access-74cbn\") pod \"4dc9dccb-784a-4082-9a7f-39223a74ce3b\" (UID: \"4dc9dccb-784a-4082-9a7f-39223a74ce3b\") "
Nov 03 20:00:24.819054 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:24.819039    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dc9dccb-784a-4082-9a7f-39223a74ce3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc9dccb-784a-4082-9a7f-39223a74ce3b-plugins-config-volume\") pod \"4dc9dccb-784a-4082-9a7f-39223a74ce3b\" (UID: \"4dc9dccb-784a-4082-9a7f-39223a74ce3b\") "
Nov 03 20:00:24.819447 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:24.819422    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/270abfe6-e1dc-4a2e-a106-4cd071d79612-plugins-config-volume" pod "270abfe6-e1dc-4a2e-a106-4cd071d79612" (UID: "270abfe6-e1dc-4a2e-a106-4cd071d79612"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:24.819542 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:24.819485    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc9dccb-784a-4082-9a7f-39223a74ce3b-plugins-config-volume" pod "4dc9dccb-784a-4082-9a7f-39223a74ce3b" (UID: "4dc9dccb-784a-4082-9a7f-39223a74ce3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:24.819790 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:24.819775    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/270abfe6-e1dc-4a2e-a106-4cd071d79612-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:24.819838 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:24.819794    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dc9dccb-784a-4082-9a7f-39223a74ce3b-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:24.821385 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:24.821351    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/270abfe6-e1dc-4a2e-a106-4cd071d79612-kube-api-access-dhtqw" pod "270abfe6-e1dc-4a2e-a106-4cd071d79612" (UID: "270abfe6-e1dc-4a2e-a106-4cd071d79612"). InnerVolumeSpecName "kube-api-access-dhtqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:24.821540 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:24.821386    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc9dccb-784a-4082-9a7f-39223a74ce3b-kube-api-access-74cbn" pod "4dc9dccb-784a-4082-9a7f-39223a74ce3b" (UID: "4dc9dccb-784a-4082-9a7f-39223a74ce3b"). InnerVolumeSpecName "kube-api-access-74cbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:24.937760 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:24.937629    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74cbn\" (UniqueName: \"kubernetes.io/projected/4dc9dccb-784a-4082-9a7f-39223a74ce3b-kube-api-access-74cbn\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:24.937760 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:24.937672    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhtqw\" (UniqueName: \"kubernetes.io/projected/270abfe6-e1dc-4a2e-a106-4cd071d79612-kube-api-access-dhtqw\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:25.003231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:25.003180    5591 generic.go:356] "Generic (PLEG): container finished" podID="270abfe6-e1dc-4a2e-a106-4cd071d79612" containerID="d8d4f00d80fb6ad4ac1f035cb304f827106df005ad0770a87a27b6d279b16fdc" exitCode=0
Nov 03 20:00:25.003413 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:25.003241    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m"
Nov 03 20:00:25.003413 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:25.003272    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m" event={"ID":"270abfe6-e1dc-4a2e-a106-4cd071d79612","Type":"ContainerDied","Data":"d8d4f00d80fb6ad4ac1f035cb304f827106df005ad0770a87a27b6d279b16fdc"}
Nov 03 20:00:25.003413 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:25.003330    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m" event={"ID":"270abfe6-e1dc-4a2e-a106-4cd071d79612","Type":"ContainerDied","Data":"c582a1ff8f1d8f9c731e3e7ed4f1b03fa6a1b7a0f0c49af17b01e7bcc26a1f17"}
Nov 03 20:00:25.003413 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:25.003352    5591 scope.go:122] "RemoveContainer" containerID="d8d4f00d80fb6ad4ac1f035cb304f827106df005ad0770a87a27b6d279b16fdc"
Nov 03 20:00:25.005061 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:25.005033    5591 generic.go:356] "Generic (PLEG): container finished" podID="4dc9dccb-784a-4082-9a7f-39223a74ce3b" containerID="935f93555316f512cb6ccfa004ba4572b8a453e30499e40c1d563f3cb977f2f0" exitCode=0
Nov 03 20:00:25.005186 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:25.005129    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68"
Nov 03 20:00:25.005308 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:25.005251    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68" event={"ID":"4dc9dccb-784a-4082-9a7f-39223a74ce3b","Type":"ContainerDied","Data":"935f93555316f512cb6ccfa004ba4572b8a453e30499e40c1d563f3cb977f2f0"}
Nov 03 20:00:25.005308 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:25.005297    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68" event={"ID":"4dc9dccb-784a-4082-9a7f-39223a74ce3b","Type":"ContainerDied","Data":"fff137e6ee0aef27271b0d80471d8b4b08c6d458ddbb7986c57caa90c8cda4af"}
Nov 03 20:00:25.015193 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:25.015174    5591 scope.go:122] "RemoveContainer" containerID="d8d4f00d80fb6ad4ac1f035cb304f827106df005ad0770a87a27b6d279b16fdc"
Nov 03 20:00:25.015539 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:00:25.015518    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8d4f00d80fb6ad4ac1f035cb304f827106df005ad0770a87a27b6d279b16fdc\": container with ID starting with d8d4f00d80fb6ad4ac1f035cb304f827106df005ad0770a87a27b6d279b16fdc not found: ID does not exist" containerID="d8d4f00d80fb6ad4ac1f035cb304f827106df005ad0770a87a27b6d279b16fdc"
Nov 03 20:00:25.015606 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:25.015549    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8d4f00d80fb6ad4ac1f035cb304f827106df005ad0770a87a27b6d279b16fdc"} err="failed to get container status \"d8d4f00d80fb6ad4ac1f035cb304f827106df005ad0770a87a27b6d279b16fdc\": rpc error: code = NotFound desc = could not find container \"d8d4f00d80fb6ad4ac1f035cb304f827106df005ad0770a87a27b6d279b16fdc\": container with ID starting with d8d4f00d80fb6ad4ac1f035cb304f827106df005ad0770a87a27b6d279b16fdc not found: ID does not exist"
Nov 03 20:00:25.015606 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:25.015568    5591 scope.go:122] "RemoveContainer" containerID="935f93555316f512cb6ccfa004ba4572b8a453e30499e40c1d563f3cb977f2f0"
Nov 03 20:00:25.028130 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:25.028108    5591 scope.go:122] "RemoveContainer" containerID="935f93555316f512cb6ccfa004ba4572b8a453e30499e40c1d563f3cb977f2f0"
Nov 03 20:00:25.028481 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:00:25.028460    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"935f93555316f512cb6ccfa004ba4572b8a453e30499e40c1d563f3cb977f2f0\": container with ID starting with 935f93555316f512cb6ccfa004ba4572b8a453e30499e40c1d563f3cb977f2f0 not found: ID does not exist" containerID="935f93555316f512cb6ccfa004ba4572b8a453e30499e40c1d563f3cb977f2f0"
Nov 03 20:00:25.028551 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:25.028494    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"935f93555316f512cb6ccfa004ba4572b8a453e30499e40c1d563f3cb977f2f0"} err="failed to get container status \"935f93555316f512cb6ccfa004ba4572b8a453e30499e40c1d563f3cb977f2f0\": rpc error: code = NotFound desc = could not find container \"935f93555316f512cb6ccfa004ba4572b8a453e30499e40c1d563f3cb977f2f0\": container with ID starting with 935f93555316f512cb6ccfa004ba4572b8a453e30499e40c1d563f3cb977f2f0 not found: ID does not exist"
Nov 03 20:00:25.034919 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:25.034889    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m"]
Nov 03 20:00:25.039629 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:25.039543    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3faa07-5c8c8448bc-9746m"]
Nov 03 20:00:25.048321 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:25.048292    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68"]
Nov 03 20:00:25.052741 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:25.052717    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3faa07-7c8cbf956-bzf68"]
Nov 03 20:00:25.354540 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:25.354510    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="270abfe6-e1dc-4a2e-a106-4cd071d79612" path="/var/lib/kubelet/pods/270abfe6-e1dc-4a2e-a106-4cd071d79612/volumes"
Nov 03 20:00:25.355055 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:25.355038    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dc9dccb-784a-4082-9a7f-39223a74ce3b" path="/var/lib/kubelet/pods/4dc9dccb-784a-4082-9a7f-39223a74ce3b/volumes"
Nov 03 20:00:26.952539 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:26.952511    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj"
Nov 03 20:00:28.738321 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:00:28.738285    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:00:32.012291 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:32.012258    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2"]
Nov 03 20:00:32.016980 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:32.016939    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz"]
Nov 03 20:00:33.626274 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:33.626237    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:00:35.907250 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:35.893826    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf"]
Nov 03 20:00:36.021158 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:36.021120    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6"]
Nov 03 20:00:37.013978 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:37.013929    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2" podUID="ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f" containerName="payload-pre-processing" containerID="cri-o://ebf7e4e956280a36943b58f09aea51c60c94cdf0538e2e590ec0f6b87fc453f8" gracePeriod=25
Nov 03 20:00:37.019367 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:37.019269    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz" podUID="e1591b82-68d7-4343-980b-8d6e1cce458c" containerName="payload-processing" containerID="cri-o://be9d8e0733db08b11a3b85739ef791c1d9036f3949518948e7492d1185d913f4" gracePeriod=25
Nov 03 20:00:37.331405 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:37.331379    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"]
Nov 03 20:00:37.331669 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:37.331643    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr" podUID="a08831b1-e194-4e39-b23d-6232d8d9851e" containerName="istio-proxy" containerID="cri-o://3cae4e3e2789caad79b9137981f7c949c98f13c5efb531f0534c6c59454011f6" gracePeriod=30
Nov 03 20:00:37.828887 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:37.828854    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2"
Nov 03 20:00:37.832825 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:37.832803    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz"
Nov 03 20:00:37.854244 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:37.854189    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1591b82-68d7-4343-980b-8d6e1cce458c-kube-api-access-7z7xm\" (UniqueName: \"kubernetes.io/projected/e1591b82-68d7-4343-980b-8d6e1cce458c-kube-api-access-7z7xm\") pod \"e1591b82-68d7-4343-980b-8d6e1cce458c\" (UID: \"e1591b82-68d7-4343-980b-8d6e1cce458c\") "
Nov 03 20:00:37.855003 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:37.854980    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f-kube-api-access-6h2qf\" (UniqueName: \"kubernetes.io/projected/ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f-kube-api-access-6h2qf\") pod \"ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f\" (UID: \"ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f\") "
Nov 03 20:00:37.855751 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:37.855733    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1591b82-68d7-4343-980b-8d6e1cce458c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1591b82-68d7-4343-980b-8d6e1cce458c-plugins-config-volume\") pod \"e1591b82-68d7-4343-980b-8d6e1cce458c\" (UID: \"e1591b82-68d7-4343-980b-8d6e1cce458c\") "
Nov 03 20:00:37.856398 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:37.856285    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1591b82-68d7-4343-980b-8d6e1cce458c-plugins-config-volume" pod "e1591b82-68d7-4343-980b-8d6e1cce458c" (UID: "e1591b82-68d7-4343-980b-8d6e1cce458c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:37.856658 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:37.856638    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f-plugins-config-volume\") pod \"ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f\" (UID: \"ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f\") "
Nov 03 20:00:37.857317 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:37.857176    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f-plugins-config-volume" pod "ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f" (UID: "ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:37.864841 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:37.864807    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1591b82-68d7-4343-980b-8d6e1cce458c-kube-api-access-7z7xm" pod "e1591b82-68d7-4343-980b-8d6e1cce458c" (UID: "e1591b82-68d7-4343-980b-8d6e1cce458c"). InnerVolumeSpecName "kube-api-access-7z7xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:37.867596 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:37.867570    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1591b82-68d7-4343-980b-8d6e1cce458c-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:37.867712 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:37.867597    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:37.867712 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:37.867612    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z7xm\" (UniqueName: \"kubernetes.io/projected/e1591b82-68d7-4343-980b-8d6e1cce458c-kube-api-access-7z7xm\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:37.867712 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:37.867677    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f-kube-api-access-6h2qf" pod "ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f" (UID: "ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f"). InnerVolumeSpecName "kube-api-access-6h2qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:37.969305 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:37.969269    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h2qf\" (UniqueName: \"kubernetes.io/projected/ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f-kube-api-access-6h2qf\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:38.148159 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:38.148121    5591 generic.go:356] "Generic (PLEG): container finished" podID="ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f" containerID="ebf7e4e956280a36943b58f09aea51c60c94cdf0538e2e590ec0f6b87fc453f8" exitCode=0
Nov 03 20:00:38.148643 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:38.148174    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2"
Nov 03 20:00:38.148643 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:38.148219    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2" event={"ID":"ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f","Type":"ContainerDied","Data":"ebf7e4e956280a36943b58f09aea51c60c94cdf0538e2e590ec0f6b87fc453f8"}
Nov 03 20:00:38.148643 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:38.148291    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2" event={"ID":"ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f","Type":"ContainerDied","Data":"c14daf634678d1fb872561ccd7dc8f4061c2850d4c35c47eb0c065cffed70d95"}
Nov 03 20:00:38.148643 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:38.148315    5591 scope.go:122] "RemoveContainer" containerID="ebf7e4e956280a36943b58f09aea51c60c94cdf0538e2e590ec0f6b87fc453f8"
Nov 03 20:00:38.149959 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:38.149792    5591 generic.go:356] "Generic (PLEG): container finished" podID="e1591b82-68d7-4343-980b-8d6e1cce458c" containerID="be9d8e0733db08b11a3b85739ef791c1d9036f3949518948e7492d1185d913f4" exitCode=0
Nov 03 20:00:38.149959 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:38.149842    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz" event={"ID":"e1591b82-68d7-4343-980b-8d6e1cce458c","Type":"ContainerDied","Data":"be9d8e0733db08b11a3b85739ef791c1d9036f3949518948e7492d1185d913f4"}
Nov 03 20:00:38.149959 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:38.149883    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz" event={"ID":"e1591b82-68d7-4343-980b-8d6e1cce458c","Type":"ContainerDied","Data":"320d7955fc0bbe8e2756b9280300fe189a32696606d07cb9c4c345f1902aad7a"}
Nov 03 20:00:38.149959 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:38.149953    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz"
Nov 03 20:00:38.161629 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:38.161603    5591 scope.go:122] "RemoveContainer" containerID="ebf7e4e956280a36943b58f09aea51c60c94cdf0538e2e590ec0f6b87fc453f8"
Nov 03 20:00:38.161841 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:00:38.161827    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebf7e4e956280a36943b58f09aea51c60c94cdf0538e2e590ec0f6b87fc453f8\": container with ID starting with ebf7e4e956280a36943b58f09aea51c60c94cdf0538e2e590ec0f6b87fc453f8 not found: ID does not exist" containerID="ebf7e4e956280a36943b58f09aea51c60c94cdf0538e2e590ec0f6b87fc453f8"
Nov 03 20:00:38.161879 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:38.161850    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf7e4e956280a36943b58f09aea51c60c94cdf0538e2e590ec0f6b87fc453f8"} err="failed to get container status \"ebf7e4e956280a36943b58f09aea51c60c94cdf0538e2e590ec0f6b87fc453f8\": rpc error: code = NotFound desc = could not find container \"ebf7e4e956280a36943b58f09aea51c60c94cdf0538e2e590ec0f6b87fc453f8\": container with ID starting with ebf7e4e956280a36943b58f09aea51c60c94cdf0538e2e590ec0f6b87fc453f8 not found: ID does not exist"
Nov 03 20:00:38.161879 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:38.161865    5591 scope.go:122] "RemoveContainer" containerID="be9d8e0733db08b11a3b85739ef791c1d9036f3949518948e7492d1185d913f4"
Nov 03 20:00:38.174700 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:38.174670    5591 scope.go:122] "RemoveContainer" containerID="be9d8e0733db08b11a3b85739ef791c1d9036f3949518948e7492d1185d913f4"
Nov 03 20:00:38.175150 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:00:38.175070    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be9d8e0733db08b11a3b85739ef791c1d9036f3949518948e7492d1185d913f4\": container with ID starting with be9d8e0733db08b11a3b85739ef791c1d9036f3949518948e7492d1185d913f4 not found: ID does not exist" containerID="be9d8e0733db08b11a3b85739ef791c1d9036f3949518948e7492d1185d913f4"
Nov 03 20:00:38.175150 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:38.175109    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9d8e0733db08b11a3b85739ef791c1d9036f3949518948e7492d1185d913f4"} err="failed to get container status \"be9d8e0733db08b11a3b85739ef791c1d9036f3949518948e7492d1185d913f4\": rpc error: code = NotFound desc = could not find container \"be9d8e0733db08b11a3b85739ef791c1d9036f3949518948e7492d1185d913f4\": container with ID starting with be9d8e0733db08b11a3b85739ef791c1d9036f3949518948e7492d1185d913f4 not found: ID does not exist"
Nov 03 20:00:38.228302 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:38.226270    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2"]
Nov 03 20:00:38.235944 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:38.235908    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8b3020-778c785455-7pqb2"]
Nov 03 20:00:38.241174 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:38.241147    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz"]
Nov 03 20:00:38.244405 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:38.244381    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8b3020-6cdd4cb6f6-v7mcz"]
Nov 03 20:00:39.352148 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:39.352116    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f" path="/var/lib/kubelet/pods/ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f/volumes"
Nov 03 20:00:39.352601 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:39.352587    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1591b82-68d7-4343-980b-8d6e1cce458c" path="/var/lib/kubelet/pods/e1591b82-68d7-4343-980b-8d6e1cce458c/volumes"
Nov 03 20:00:39.737969 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:39.737813    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8"]
Nov 03 20:00:39.738326 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:39.738045    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8" podUID="12d2b7e4-96b2-4ee9-a511-4ca0f21228fe" containerName="maas-api" containerID="cri-o://c9bb286110ece406e2f59a98d9ba3c6505a2bf9ddfdf152642718d16e2f14e89" gracePeriod=30
Nov 03 20:00:40.194233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:40.186948    5591 generic.go:356] "Generic (PLEG): container finished" podID="12d2b7e4-96b2-4ee9-a511-4ca0f21228fe" containerID="c9bb286110ece406e2f59a98d9ba3c6505a2bf9ddfdf152642718d16e2f14e89" exitCode=0
Nov 03 20:00:40.194233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:40.187105    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8" event={"ID":"12d2b7e4-96b2-4ee9-a511-4ca0f21228fe","Type":"ContainerDied","Data":"c9bb286110ece406e2f59a98d9ba3c6505a2bf9ddfdf152642718d16e2f14e89"}
Nov 03 20:00:40.740089 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:40.739768    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8"
Nov 03 20:00:40.834019 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:40.833977    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-metrics-certs\") pod \"12d2b7e4-96b2-4ee9-a511-4ca0f21228fe\" (UID: \"12d2b7e4-96b2-4ee9-a511-4ca0f21228fe\") "
Nov 03 20:00:40.834221 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:40.834099    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-maas-api-tls\") pod \"12d2b7e4-96b2-4ee9-a511-4ca0f21228fe\" (UID: \"12d2b7e4-96b2-4ee9-a511-4ca0f21228fe\") "
Nov 03 20:00:40.834221 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:40.834176    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-kube-api-access-8hcb9\" (UniqueName: \"kubernetes.io/projected/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-kube-api-access-8hcb9\") pod \"12d2b7e4-96b2-4ee9-a511-4ca0f21228fe\" (UID: \"12d2b7e4-96b2-4ee9-a511-4ca0f21228fe\") "
Nov 03 20:00:40.834382 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:40.834277    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-openshift-service-ca\") pod \"12d2b7e4-96b2-4ee9-a511-4ca0f21228fe\" (UID: \"12d2b7e4-96b2-4ee9-a511-4ca0f21228fe\") "
Nov 03 20:00:40.835337 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:40.835313    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-openshift-service-ca" pod "12d2b7e4-96b2-4ee9-a511-4ca0f21228fe" (UID: "12d2b7e4-96b2-4ee9-a511-4ca0f21228fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:40.837634 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:40.837599    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-maas-api-tls" pod "12d2b7e4-96b2-4ee9-a511-4ca0f21228fe" (UID: "12d2b7e4-96b2-4ee9-a511-4ca0f21228fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:00:40.838342 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:40.837799    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-metrics-certs" pod "12d2b7e4-96b2-4ee9-a511-4ca0f21228fe" (UID: "12d2b7e4-96b2-4ee9-a511-4ca0f21228fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:00:40.838342 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:40.838140    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-kube-api-access-8hcb9" pod "12d2b7e4-96b2-4ee9-a511-4ca0f21228fe" (UID: "12d2b7e4-96b2-4ee9-a511-4ca0f21228fe"). InnerVolumeSpecName "kube-api-access-8hcb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:40.910375 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:40.910276    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf" podUID="3efe99ef-4e27-4f9c-994b-ebdd23792b8c" containerName="payload-processing" containerID="cri-o://5909ab93566cf248a5770b3b87c73c397def596e09635a91376a42fb74862800" gracePeriod=25
Nov 03 20:00:40.947026 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:40.946905    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:40.947026 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:40.946937    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:40.947026 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:40.946950    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hcb9\" (UniqueName: \"kubernetes.io/projected/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-kube-api-access-8hcb9\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:40.947026 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:40.946963    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:41.025714 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:41.025420    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6" podUID="43ad69e4-a321-4324-9e2a-9789fed758e9" containerName="payload-pre-processing" containerID="cri-o://14fca4a8d55eda6e007e9c889f97fa769ad47a63ee7a876071c46fe658103c47" gracePeriod=25
Nov 03 20:00:41.066485 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:00:41.066459    5591 logging.go:55] [core] [Channel #1913 SubChannel #1914]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.145:50052", ServerName: "10.134.0.145:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.145:50052: connect: connection refused"
Nov 03 20:00:41.226012 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:41.225913    5591 generic.go:356] "Generic (PLEG): container finished" podID="3efe99ef-4e27-4f9c-994b-ebdd23792b8c" containerID="5909ab93566cf248a5770b3b87c73c397def596e09635a91376a42fb74862800" exitCode=0
Nov 03 20:00:41.226398 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:41.226358    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf" event={"ID":"3efe99ef-4e27-4f9c-994b-ebdd23792b8c","Type":"ContainerDied","Data":"5909ab93566cf248a5770b3b87c73c397def596e09635a91376a42fb74862800"}
Nov 03 20:00:41.237103 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:41.237068    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8" event={"ID":"12d2b7e4-96b2-4ee9-a511-4ca0f21228fe","Type":"ContainerDied","Data":"329057f5c491c8b0a9b6932d69decd31ae5ae18efe34d957490b9452c96a083d"}
Nov 03 20:00:41.237474 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:41.237080    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8"
Nov 03 20:00:41.239878 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:41.239489    5591 scope.go:122] "RemoveContainer" containerID="c9bb286110ece406e2f59a98d9ba3c6505a2bf9ddfdf152642718d16e2f14e89"
Nov 03 20:00:41.240083 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:41.240057    5591 generic.go:356] "Generic (PLEG): container finished" podID="43ad69e4-a321-4324-9e2a-9789fed758e9" containerID="14fca4a8d55eda6e007e9c889f97fa769ad47a63ee7a876071c46fe658103c47" exitCode=0
Nov 03 20:00:41.240237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:41.240107    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6" event={"ID":"43ad69e4-a321-4324-9e2a-9789fed758e9","Type":"ContainerDied","Data":"14fca4a8d55eda6e007e9c889f97fa769ad47a63ee7a876071c46fe658103c47"}
Nov 03 20:00:41.300834 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:41.300791    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8"]
Nov 03 20:00:41.313483 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:41.313449    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5cf30a-f46d848bd-jn9g8"]
Nov 03 20:00:41.355557 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:41.355402    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12d2b7e4-96b2-4ee9-a511-4ca0f21228fe" path="/var/lib/kubelet/pods/12d2b7e4-96b2-4ee9-a511-4ca0f21228fe/volumes"
Nov 03 20:00:41.554787 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:41.554749    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf"
Nov 03 20:00:41.559313 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:41.559284    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6"
Nov 03 20:00:41.559579 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:41.559558    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3efe99ef-4e27-4f9c-994b-ebdd23792b8c-kube-api-access-lf2vl\" (UniqueName: \"kubernetes.io/projected/3efe99ef-4e27-4f9c-994b-ebdd23792b8c-kube-api-access-lf2vl\") pod \"3efe99ef-4e27-4f9c-994b-ebdd23792b8c\" (UID: \"3efe99ef-4e27-4f9c-994b-ebdd23792b8c\") "
Nov 03 20:00:41.559865 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:41.559809    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3efe99ef-4e27-4f9c-994b-ebdd23792b8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3efe99ef-4e27-4f9c-994b-ebdd23792b8c-plugins-config-volume\") pod \"3efe99ef-4e27-4f9c-994b-ebdd23792b8c\" (UID: \"3efe99ef-4e27-4f9c-994b-ebdd23792b8c\") "
Nov 03 20:00:41.560666 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:41.560640    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3efe99ef-4e27-4f9c-994b-ebdd23792b8c-plugins-config-volume" pod "3efe99ef-4e27-4f9c-994b-ebdd23792b8c" (UID: "3efe99ef-4e27-4f9c-994b-ebdd23792b8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:41.562709 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:41.562678    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3efe99ef-4e27-4f9c-994b-ebdd23792b8c-kube-api-access-lf2vl" pod "3efe99ef-4e27-4f9c-994b-ebdd23792b8c" (UID: "3efe99ef-4e27-4f9c-994b-ebdd23792b8c"). InnerVolumeSpecName "kube-api-access-lf2vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:41.661428 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:41.661398    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ad69e4-a321-4324-9e2a-9789fed758e9-kube-api-access-kmnnc\" (UniqueName: \"kubernetes.io/projected/43ad69e4-a321-4324-9e2a-9789fed758e9-kube-api-access-kmnnc\") pod \"43ad69e4-a321-4324-9e2a-9789fed758e9\" (UID: \"43ad69e4-a321-4324-9e2a-9789fed758e9\") "
Nov 03 20:00:41.661651 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:41.661461    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ad69e4-a321-4324-9e2a-9789fed758e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ad69e4-a321-4324-9e2a-9789fed758e9-plugins-config-volume\") pod \"43ad69e4-a321-4324-9e2a-9789fed758e9\" (UID: \"43ad69e4-a321-4324-9e2a-9789fed758e9\") "
Nov 03 20:00:41.661892 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:41.661877    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3efe99ef-4e27-4f9c-994b-ebdd23792b8c-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:41.662287 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:41.661910    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf2vl\" (UniqueName: \"kubernetes.io/projected/3efe99ef-4e27-4f9c-994b-ebdd23792b8c-kube-api-access-lf2vl\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:41.662498 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:41.662410    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ad69e4-a321-4324-9e2a-9789fed758e9-plugins-config-volume" pod "43ad69e4-a321-4324-9e2a-9789fed758e9" (UID: "43ad69e4-a321-4324-9e2a-9789fed758e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:41.664832 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:41.664809    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ad69e4-a321-4324-9e2a-9789fed758e9-kube-api-access-kmnnc" pod "43ad69e4-a321-4324-9e2a-9789fed758e9" (UID: "43ad69e4-a321-4324-9e2a-9789fed758e9"). InnerVolumeSpecName "kube-api-access-kmnnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:41.763590 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:41.763555    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmnnc\" (UniqueName: \"kubernetes.io/projected/43ad69e4-a321-4324-9e2a-9789fed758e9-kube-api-access-kmnnc\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:41.763590 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:41.763584    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ad69e4-a321-4324-9e2a-9789fed758e9-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:42.066550 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.066491    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf" podUID="3efe99ef-4e27-4f9c-994b-ebdd23792b8c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.145:50052\" within 1s: context deadline exceeded"
Nov 03 20:00:42.251162 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.251076    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6"
Nov 03 20:00:42.251162 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.251115    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6" event={"ID":"43ad69e4-a321-4324-9e2a-9789fed758e9","Type":"ContainerDied","Data":"a7c9eac56d1472bc578b99577b1900cfdb8ad992852c4e8933ce3f2be891bf92"}
Nov 03 20:00:42.251442 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.251179    5591 scope.go:122] "RemoveContainer" containerID="14fca4a8d55eda6e007e9c889f97fa769ad47a63ee7a876071c46fe658103c47"
Nov 03 20:00:42.253104 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.253083    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf" event={"ID":"3efe99ef-4e27-4f9c-994b-ebdd23792b8c","Type":"ContainerDied","Data":"1868c3eeb00e73e0e736ab2e2cbf62c55f2ea53cebbf3685f4ed931c1ad07458"}
Nov 03 20:00:42.253272 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.253113    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf"
Nov 03 20:00:42.315221 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.312390    5591 scope.go:122] "RemoveContainer" containerID="5909ab93566cf248a5770b3b87c73c397def596e09635a91376a42fb74862800"
Nov 03 20:00:42.328430 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.323109    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6"]
Nov 03 20:00:42.407976 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.407905    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-5cf30a-685bdb4b4c-47jq6"]
Nov 03 20:00:42.417434 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.417371    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf"]
Nov 03 20:00:42.429406 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.419763    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-5cf30a-95c8ff4c8-7hjcf"]
Nov 03 20:00:42.675691 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.675662    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 20:00:42.717616 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.717587    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-credential-socket\") pod \"a08831b1-e194-4e39-b23d-6232d8d9851e\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") "
Nov 03 20:00:42.721584 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.721336    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-envoy\") pod \"a08831b1-e194-4e39-b23d-6232d8d9851e\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") "
Nov 03 20:00:42.721584 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.721421    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-podinfo\") pod \"a08831b1-e194-4e39-b23d-6232d8d9851e\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") "
Nov 03 20:00:42.721584 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.721495    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a08831b1-e194-4e39-b23d-6232d8d9851e-kube-api-access-2k48j\" (UniqueName: \"kubernetes.io/projected/a08831b1-e194-4e39-b23d-6232d8d9851e-kube-api-access-2k48j\") pod \"a08831b1-e194-4e39-b23d-6232d8d9851e\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") "
Nov 03 20:00:42.721584 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.721539    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-token\" (UniqueName: \"kubernetes.io/projected/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-token\") pod \"a08831b1-e194-4e39-b23d-6232d8d9851e\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") "
Nov 03 20:00:42.721870 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.721595    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-data\") pod \"a08831b1-e194-4e39-b23d-6232d8d9851e\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") "
Nov 03 20:00:42.721870 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.721625    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a08831b1-e194-4e39-b23d-6232d8d9851e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a08831b1-e194-4e39-b23d-6232d8d9851e-istiod-ca-cert\") pod \"a08831b1-e194-4e39-b23d-6232d8d9851e\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") "
Nov 03 20:00:42.721870 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.717937    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-credential-socket" pod "a08831b1-e194-4e39-b23d-6232d8d9851e" (UID: "a08831b1-e194-4e39-b23d-6232d8d9851e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:00:42.721870 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.721709    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-workload-socket\") pod \"a08831b1-e194-4e39-b23d-6232d8d9851e\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") "
Nov 03 20:00:42.721870 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.721767    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-workload-certs\") pod \"a08831b1-e194-4e39-b23d-6232d8d9851e\" (UID: \"a08831b1-e194-4e39-b23d-6232d8d9851e\") "
Nov 03 20:00:42.722664 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.722486    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:42.722664 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.722497    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a08831b1-e194-4e39-b23d-6232d8d9851e-istiod-ca-cert" pod "a08831b1-e194-4e39-b23d-6232d8d9851e" (UID: "a08831b1-e194-4e39-b23d-6232d8d9851e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:42.722921 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.722690    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-workload-certs" pod "a08831b1-e194-4e39-b23d-6232d8d9851e" (UID: "a08831b1-e194-4e39-b23d-6232d8d9851e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:00:42.723076 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.723029    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-workload-socket" pod "a08831b1-e194-4e39-b23d-6232d8d9851e" (UID: "a08831b1-e194-4e39-b23d-6232d8d9851e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:00:42.725126 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.725061    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-data" pod "a08831b1-e194-4e39-b23d-6232d8d9851e" (UID: "a08831b1-e194-4e39-b23d-6232d8d9851e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:00:42.726809 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.726774    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-podinfo" pod "a08831b1-e194-4e39-b23d-6232d8d9851e" (UID: "a08831b1-e194-4e39-b23d-6232d8d9851e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:00:42.727433 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.727406    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-token" pod "a08831b1-e194-4e39-b23d-6232d8d9851e" (UID: "a08831b1-e194-4e39-b23d-6232d8d9851e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:42.728372 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.728335    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a08831b1-e194-4e39-b23d-6232d8d9851e-kube-api-access-2k48j" pod "a08831b1-e194-4e39-b23d-6232d8d9851e" (UID: "a08831b1-e194-4e39-b23d-6232d8d9851e"). InnerVolumeSpecName "kube-api-access-2k48j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:42.738728 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.738691    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-envoy" pod "a08831b1-e194-4e39-b23d-6232d8d9851e" (UID: "a08831b1-e194-4e39-b23d-6232d8d9851e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:00:42.824016 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.823921    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k48j\" (UniqueName: \"kubernetes.io/projected/a08831b1-e194-4e39-b23d-6232d8d9851e-kube-api-access-2k48j\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:42.824016 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.823962    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:42.824016 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.823978    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:42.824016 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.823991    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a08831b1-e194-4e39-b23d-6232d8d9851e-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:42.824016 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.824007    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:42.824016 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.824021    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:42.824822 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.824035    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:42.824822 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:42.824052    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a08831b1-e194-4e39-b23d-6232d8d9851e-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:43.262384 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:43.262291    5591 generic.go:356] "Generic (PLEG): container finished" podID="a08831b1-e194-4e39-b23d-6232d8d9851e" containerID="3cae4e3e2789caad79b9137981f7c949c98f13c5efb531f0534c6c59454011f6" exitCode=0
Nov 03 20:00:43.262384 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:43.262358    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"
Nov 03 20:00:43.262625 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:43.262382    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr" event={"ID":"a08831b1-e194-4e39-b23d-6232d8d9851e","Type":"ContainerDied","Data":"3cae4e3e2789caad79b9137981f7c949c98f13c5efb531f0534c6c59454011f6"}
Nov 03 20:00:43.262625 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:43.262447    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr" event={"ID":"a08831b1-e194-4e39-b23d-6232d8d9851e","Type":"ContainerDied","Data":"8e4bbbbe1d8c6064741e6eb9cfd7122a6726052a54b2e773a023f2567c1c019d"}
Nov 03 20:00:43.262625 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:43.262480    5591 scope.go:122] "RemoveContainer" containerID="3cae4e3e2789caad79b9137981f7c949c98f13c5efb531f0534c6c59454011f6"
Nov 03 20:00:43.310441 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:43.310344    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"]
Nov 03 20:00:43.311334 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:43.311242    5591 scope.go:122] "RemoveContainer" containerID="3cae4e3e2789caad79b9137981f7c949c98f13c5efb531f0534c6c59454011f6"
Nov 03 20:00:43.311726 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:00:43.311678    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cae4e3e2789caad79b9137981f7c949c98f13c5efb531f0534c6c59454011f6\": container with ID starting with 3cae4e3e2789caad79b9137981f7c949c98f13c5efb531f0534c6c59454011f6 not found: ID does not exist" containerID="3cae4e3e2789caad79b9137981f7c949c98f13c5efb531f0534c6c59454011f6"
Nov 03 20:00:43.311726 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:43.311705    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cae4e3e2789caad79b9137981f7c949c98f13c5efb531f0534c6c59454011f6"} err="failed to get container status \"3cae4e3e2789caad79b9137981f7c949c98f13c5efb531f0534c6c59454011f6\": rpc error: code = NotFound desc = could not find container \"3cae4e3e2789caad79b9137981f7c949c98f13c5efb531f0534c6c59454011f6\": container with ID starting with 3cae4e3e2789caad79b9137981f7c949c98f13c5efb531f0534c6c59454011f6 not found: ID does not exist"
Nov 03 20:00:43.328423 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:43.328389    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-389e6d-openshift-default-799f7cdcdf-lj8xr"]
Nov 03 20:00:43.351915 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:43.351885    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3efe99ef-4e27-4f9c-994b-ebdd23792b8c" path="/var/lib/kubelet/pods/3efe99ef-4e27-4f9c-994b-ebdd23792b8c/volumes"
Nov 03 20:00:43.352498 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:43.352480    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43ad69e4-a321-4324-9e2a-9789fed758e9" path="/var/lib/kubelet/pods/43ad69e4-a321-4324-9e2a-9789fed758e9/volumes"
Nov 03 20:00:43.352995 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:43.352979    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a08831b1-e194-4e39-b23d-6232d8d9851e" path="/var/lib/kubelet/pods/a08831b1-e194-4e39-b23d-6232d8d9851e/volumes"
Nov 03 20:00:43.672942 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:43.672884    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"]
Nov 03 20:00:43.673464 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:43.673432    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz" podUID="928410e8-ddb8-4a61-8a41-f218603c0e47" containerName="istio-proxy" containerID="cri-o://51ca72b4a5ae046f96e4fd65781a3cd4a97f7078c6ab97ca0415efd732c3f754" gracePeriod=30
Nov 03 20:00:47.343123 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:47.343083    5591 generic.go:356] "Generic (PLEG): container finished" podID="56528ce3-95f8-408d-96db-04ac7bcab926" containerID="bc51a650762e67653343b8311b56e1151c2fbe39cb177e5f812fc9753ed3be0e" exitCode=0
Nov 03 20:00:47.349723 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:47.349694    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x" event={"ID":"56528ce3-95f8-408d-96db-04ac7bcab926","Type":"ContainerDied","Data":"bc51a650762e67653343b8311b56e1151c2fbe39cb177e5f812fc9753ed3be0e"}
Nov 03 20:00:47.350168 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:47.350151    5591 scope.go:122] "RemoveContainer" containerID="bc51a650762e67653343b8311b56e1151c2fbe39cb177e5f812fc9753ed3be0e"
Nov 03 20:00:47.351779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:47.351749    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:00:47.351848 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:47.351821    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:00:47.351886 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:47.351854    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:00:48.353602 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:48.353563    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x" event={"ID":"56528ce3-95f8-408d-96db-04ac7bcab926","Type":"ContainerStarted","Data":"3b4385c98ae8bb6442e29c7c255251d46546c8adb221f4cacd1450be56f406f5"}
Nov 03 20:00:48.382235 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:48.381678    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:48.382235 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:48.381848    5591 patch_prober.go:29] interesting pod/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:00:48.382235 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:48.381927    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x" podUID="56528ce3-95f8-408d-96db-04ac7bcab926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 03 20:00:48.671093 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:48.671052    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v"]
Nov 03 20:00:48.672469 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:48.672359    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v" podUID="bf807ba9-d308-4b56-96a4-702135b4d522" containerName="maas-api" containerID="cri-o://67258483a817d28c97e180d083219eafcc8ba25241810ac5a93741c370d810c3" gracePeriod=30
Nov 03 20:00:48.818191 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:48.817043    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5"]
Nov 03 20:00:48.818191 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:48.817310    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5" podUID="09c9289a-f596-4c46-b4a0-a85cc1cc70e5" containerName="payload-pre-processing" containerID="cri-o://b10f53fa7418fd888b875d696e0f2f0bd04e2a81946b57498b329d2d513b1229" gracePeriod=30
Nov 03 20:00:49.428847 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.391781    5591 patch_prober.go:29] interesting pod/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:00:49.428847 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.391840    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x" podUID="56528ce3-95f8-408d-96db-04ac7bcab926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 03 20:00:49.461155 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.460225    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v"
Nov 03 20:00:49.481852 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.476280    5591 generic.go:356] "Generic (PLEG): container finished" podID="928410e8-ddb8-4a61-8a41-f218603c0e47" containerID="51ca72b4a5ae046f96e4fd65781a3cd4a97f7078c6ab97ca0415efd732c3f754" exitCode=0
Nov 03 20:00:49.600811 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.600252    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf807ba9-d308-4b56-96a4-702135b4d522-kube-api-access-jghjp\" (UniqueName: \"kubernetes.io/projected/bf807ba9-d308-4b56-96a4-702135b4d522-kube-api-access-jghjp\") pod \"bf807ba9-d308-4b56-96a4-702135b4d522\" (UID: \"bf807ba9-d308-4b56-96a4-702135b4d522\") "
Nov 03 20:00:49.600811 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.600433    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf807ba9-d308-4b56-96a4-702135b4d522-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf807ba9-d308-4b56-96a4-702135b4d522-openshift-service-ca\") pod \"bf807ba9-d308-4b56-96a4-702135b4d522\" (UID: \"bf807ba9-d308-4b56-96a4-702135b4d522\") "
Nov 03 20:00:49.600811 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.600657    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf807ba9-d308-4b56-96a4-702135b4d522-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf807ba9-d308-4b56-96a4-702135b4d522-maas-api-tls\") pod \"bf807ba9-d308-4b56-96a4-702135b4d522\" (UID: \"bf807ba9-d308-4b56-96a4-702135b4d522\") "
Nov 03 20:00:49.600811 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.600750    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf807ba9-d308-4b56-96a4-702135b4d522-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf807ba9-d308-4b56-96a4-702135b4d522-metrics-certs\") pod \"bf807ba9-d308-4b56-96a4-702135b4d522\" (UID: \"bf807ba9-d308-4b56-96a4-702135b4d522\") "
Nov 03 20:00:49.616612 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.616230    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf807ba9-d308-4b56-96a4-702135b4d522-openshift-service-ca" pod "bf807ba9-d308-4b56-96a4-702135b4d522" (UID: "bf807ba9-d308-4b56-96a4-702135b4d522"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:49.627601 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.627541    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf807ba9-d308-4b56-96a4-702135b4d522-kube-api-access-jghjp" pod "bf807ba9-d308-4b56-96a4-702135b4d522" (UID: "bf807ba9-d308-4b56-96a4-702135b4d522"). InnerVolumeSpecName "kube-api-access-jghjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:49.666225 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.666026    5591 generic.go:356] "Generic (PLEG): container finished" podID="bf807ba9-d308-4b56-96a4-702135b4d522" containerID="67258483a817d28c97e180d083219eafcc8ba25241810ac5a93741c370d810c3" exitCode=0
Nov 03 20:00:49.666225 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.666225    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v"
Nov 03 20:00:49.702664 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.690754    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf807ba9-d308-4b56-96a4-702135b4d522-metrics-certs" pod "bf807ba9-d308-4b56-96a4-702135b4d522" (UID: "bf807ba9-d308-4b56-96a4-702135b4d522"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:00:49.731365 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.724527    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf807ba9-d308-4b56-96a4-702135b4d522-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:49.731365 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.724558    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jghjp\" (UniqueName: \"kubernetes.io/projected/bf807ba9-d308-4b56-96a4-702135b4d522-kube-api-access-jghjp\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:49.731365 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.724574    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf807ba9-d308-4b56-96a4-702135b4d522-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:49.731365 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.724862    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf807ba9-d308-4b56-96a4-702135b4d522-maas-api-tls" pod "bf807ba9-d308-4b56-96a4-702135b4d522" (UID: "bf807ba9-d308-4b56-96a4-702135b4d522"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:00:49.731365 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.725647    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz" event={"ID":"928410e8-ddb8-4a61-8a41-f218603c0e47","Type":"ContainerDied","Data":"51ca72b4a5ae046f96e4fd65781a3cd4a97f7078c6ab97ca0415efd732c3f754"}
Nov 03 20:00:49.731365 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.725706    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v" event={"ID":"bf807ba9-d308-4b56-96a4-702135b4d522","Type":"ContainerDied","Data":"67258483a817d28c97e180d083219eafcc8ba25241810ac5a93741c370d810c3"}
Nov 03 20:00:49.731365 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.725733    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v" event={"ID":"bf807ba9-d308-4b56-96a4-702135b4d522","Type":"ContainerDied","Data":"93abca9a463edd18471d41065b7b454288b94bcc58182c242319dbeb2f837728"}
Nov 03 20:00:49.731365 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.725759    5591 scope.go:122] "RemoveContainer" containerID="67258483a817d28c97e180d083219eafcc8ba25241810ac5a93741c370d810c3"
Nov 03 20:00:49.731365 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.727576    5591 generic.go:356] "Generic (PLEG): container finished" podID="09c9289a-f596-4c46-b4a0-a85cc1cc70e5" containerID="b10f53fa7418fd888b875d696e0f2f0bd04e2a81946b57498b329d2d513b1229" exitCode=0
Nov 03 20:00:49.731365 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.728304    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5" event={"ID":"09c9289a-f596-4c46-b4a0-a85cc1cc70e5","Type":"ContainerDied","Data":"b10f53fa7418fd888b875d696e0f2f0bd04e2a81946b57498b329d2d513b1229"}
Nov 03 20:00:49.743802 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.743772    5591 scope.go:122] "RemoveContainer" containerID="67258483a817d28c97e180d083219eafcc8ba25241810ac5a93741c370d810c3"
Nov 03 20:00:49.744184 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:00:49.744160    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67258483a817d28c97e180d083219eafcc8ba25241810ac5a93741c370d810c3\": container with ID starting with 67258483a817d28c97e180d083219eafcc8ba25241810ac5a93741c370d810c3 not found: ID does not exist" containerID="67258483a817d28c97e180d083219eafcc8ba25241810ac5a93741c370d810c3"
Nov 03 20:00:49.744938 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.744906    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67258483a817d28c97e180d083219eafcc8ba25241810ac5a93741c370d810c3"} err="failed to get container status \"67258483a817d28c97e180d083219eafcc8ba25241810ac5a93741c370d810c3\": rpc error: code = NotFound desc = could not find container \"67258483a817d28c97e180d083219eafcc8ba25241810ac5a93741c370d810c3\": container with ID starting with 67258483a817d28c97e180d083219eafcc8ba25241810ac5a93741c370d810c3 not found: ID does not exist"
Nov 03 20:00:49.826995 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.826961    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf807ba9-d308-4b56-96a4-702135b4d522-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:49.853562 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.853481    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5"
Nov 03 20:00:49.922321 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.922293    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 20:00:49.927870 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.927842    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09c9289a-f596-4c46-b4a0-a85cc1cc70e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09c9289a-f596-4c46-b4a0-a85cc1cc70e5-plugins-config-volume\") pod \"09c9289a-f596-4c46-b4a0-a85cc1cc70e5\" (UID: \"09c9289a-f596-4c46-b4a0-a85cc1cc70e5\") "
Nov 03 20:00:49.928119 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.928103    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c9289a-f596-4c46-b4a0-a85cc1cc70e5-kube-api-access-9dqjj\" (UniqueName: \"kubernetes.io/projected/09c9289a-f596-4c46-b4a0-a85cc1cc70e5-kube-api-access-9dqjj\") pod \"09c9289a-f596-4c46-b4a0-a85cc1cc70e5\" (UID: \"09c9289a-f596-4c46-b4a0-a85cc1cc70e5\") "
Nov 03 20:00:49.928265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.928238    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c9289a-f596-4c46-b4a0-a85cc1cc70e5-plugins-config-volume" pod "09c9289a-f596-4c46-b4a0-a85cc1cc70e5" (UID: "09c9289a-f596-4c46-b4a0-a85cc1cc70e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:49.928765 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.928733    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09c9289a-f596-4c46-b4a0-a85cc1cc70e5-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:49.931313 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:49.931292    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c9289a-f596-4c46-b4a0-a85cc1cc70e5-kube-api-access-9dqjj" pod "09c9289a-f596-4c46-b4a0-a85cc1cc70e5" (UID: "09c9289a-f596-4c46-b4a0-a85cc1cc70e5"). InnerVolumeSpecName "kube-api-access-9dqjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:50.008645 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.008606    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v"]
Nov 03 20:00:50.012274 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.012242    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3d1db1-799fd9b5d7-6d54v"]
Nov 03 20:00:50.029620 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.029588    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-podinfo\") pod \"928410e8-ddb8-4a61-8a41-f218603c0e47\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") "
Nov 03 20:00:50.029793 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.029689    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-workload-certs\") pod \"928410e8-ddb8-4a61-8a41-f218603c0e47\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") "
Nov 03 20:00:50.029793 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.029712    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-envoy\") pod \"928410e8-ddb8-4a61-8a41-f218603c0e47\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") "
Nov 03 20:00:50.029793 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.029766    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/928410e8-ddb8-4a61-8a41-f218603c0e47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/928410e8-ddb8-4a61-8a41-f218603c0e47-istiod-ca-cert\") pod \"928410e8-ddb8-4a61-8a41-f218603c0e47\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") "
Nov 03 20:00:50.029793 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.029792    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/928410e8-ddb8-4a61-8a41-f218603c0e47-kube-api-access-gzrcb\" (UniqueName: \"kubernetes.io/projected/928410e8-ddb8-4a61-8a41-f218603c0e47-kube-api-access-gzrcb\") pod \"928410e8-ddb8-4a61-8a41-f218603c0e47\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") "
Nov 03 20:00:50.030433 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.029818    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-token\" (UniqueName: \"kubernetes.io/projected/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-token\") pod \"928410e8-ddb8-4a61-8a41-f218603c0e47\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") "
Nov 03 20:00:50.030433 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.029842    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-credential-socket\") pod \"928410e8-ddb8-4a61-8a41-f218603c0e47\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") "
Nov 03 20:00:50.030433 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.029866    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-workload-socket\") pod \"928410e8-ddb8-4a61-8a41-f218603c0e47\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") "
Nov 03 20:00:50.030433 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.029939    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-data\") pod \"928410e8-ddb8-4a61-8a41-f218603c0e47\" (UID: \"928410e8-ddb8-4a61-8a41-f218603c0e47\") "
Nov 03 20:00:50.030433 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.029985    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-workload-certs" pod "928410e8-ddb8-4a61-8a41-f218603c0e47" (UID: "928410e8-ddb8-4a61-8a41-f218603c0e47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:00:50.030433 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.030131    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-credential-socket" pod "928410e8-ddb8-4a61-8a41-f218603c0e47" (UID: "928410e8-ddb8-4a61-8a41-f218603c0e47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:00:50.030433 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.030172    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-workload-socket" pod "928410e8-ddb8-4a61-8a41-f218603c0e47" (UID: "928410e8-ddb8-4a61-8a41-f218603c0e47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:00:50.030433 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.030322    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-data" pod "928410e8-ddb8-4a61-8a41-f218603c0e47" (UID: "928410e8-ddb8-4a61-8a41-f218603c0e47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:00:50.030885 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.030492    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/928410e8-ddb8-4a61-8a41-f218603c0e47-istiod-ca-cert" pod "928410e8-ddb8-4a61-8a41-f218603c0e47" (UID: "928410e8-ddb8-4a61-8a41-f218603c0e47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:00:50.030885 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.030743    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:50.030885 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.030766    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dqjj\" (UniqueName: \"kubernetes.io/projected/09c9289a-f596-4c46-b4a0-a85cc1cc70e5-kube-api-access-9dqjj\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:50.030885 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.030782    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/928410e8-ddb8-4a61-8a41-f218603c0e47-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:50.030885 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.030796    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:50.030885 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.030810    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:50.030885 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.030825    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:50.032063 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.032042    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-envoy" pod "928410e8-ddb8-4a61-8a41-f218603c0e47" (UID: "928410e8-ddb8-4a61-8a41-f218603c0e47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:00:50.032333 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.032313    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/928410e8-ddb8-4a61-8a41-f218603c0e47-kube-api-access-gzrcb" pod "928410e8-ddb8-4a61-8a41-f218603c0e47" (UID: "928410e8-ddb8-4a61-8a41-f218603c0e47"). InnerVolumeSpecName "kube-api-access-gzrcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:50.032593 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.032569    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-podinfo" pod "928410e8-ddb8-4a61-8a41-f218603c0e47" (UID: "928410e8-ddb8-4a61-8a41-f218603c0e47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:00:50.032686 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.032618    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-token" pod "928410e8-ddb8-4a61-8a41-f218603c0e47" (UID: "928410e8-ddb8-4a61-8a41-f218603c0e47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:00:50.132452 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.132371    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:50.132452 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.132399    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzrcb\" (UniqueName: \"kubernetes.io/projected/928410e8-ddb8-4a61-8a41-f218603c0e47-kube-api-access-gzrcb\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:50.132452 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.132410    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:50.132452 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.132419    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/928410e8-ddb8-4a61-8a41-f218603c0e47-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:00:50.401594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.401370    5591 patch_prober.go:29] interesting pod/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:00:50.401594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.401453    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x" podUID="56528ce3-95f8-408d-96db-04ac7bcab926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 03 20:00:50.543594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.543555    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:50.806324 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.796386    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz" event={"ID":"928410e8-ddb8-4a61-8a41-f218603c0e47","Type":"ContainerDied","Data":"e15c6b3ed45cb8bdc07eedf23c13decfa7343203ce9f6f6475fc49c7e7901c84"}
Nov 03 20:00:50.806324 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.796464    5591 scope.go:122] "RemoveContainer" containerID="51ca72b4a5ae046f96e4fd65781a3cd4a97f7078c6ab97ca0415efd732c3f754"
Nov 03 20:00:50.806324 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.796568    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"
Nov 03 20:00:50.811859 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.811830    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5" event={"ID":"09c9289a-f596-4c46-b4a0-a85cc1cc70e5","Type":"ContainerDied","Data":"06918d60b14722d2e9623471452e39df3f73eab21330d78370311567922c082a"}
Nov 03 20:00:50.812032 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.811906    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5"
Nov 03 20:00:50.819512 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.819376    5591 scope.go:122] "RemoveContainer" containerID="b10f53fa7418fd888b875d696e0f2f0bd04e2a81946b57498b329d2d513b1229"
Nov 03 20:00:50.837841 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.837815    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"]
Nov 03 20:00:50.843067 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.843039    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-5cf30a-openshift-default-64bb6488b5-999mz"]
Nov 03 20:00:50.856953 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.856923    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5"]
Nov 03 20:00:50.861279 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:50.861254    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3d1db1-57776c5455-htkh5"]
Nov 03 20:00:51.351364 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:51.351333    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09c9289a-f596-4c46-b4a0-a85cc1cc70e5" path="/var/lib/kubelet/pods/09c9289a-f596-4c46-b4a0-a85cc1cc70e5/volumes"
Nov 03 20:00:51.351938 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:51.351914    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="928410e8-ddb8-4a61-8a41-f218603c0e47" path="/var/lib/kubelet/pods/928410e8-ddb8-4a61-8a41-f218603c0e47/volumes"
Nov 03 20:00:51.352537 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:51.352509    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf807ba9-d308-4b56-96a4-702135b4d522" path="/var/lib/kubelet/pods/bf807ba9-d308-4b56-96a4-702135b4d522/volumes"
Nov 03 20:00:51.395233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:51.376330    5591 patch_prober.go:29] interesting pod/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:00:51.395233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:51.376385    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x" podUID="56528ce3-95f8-408d-96db-04ac7bcab926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 03 20:00:52.389232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:52.383404    5591 patch_prober.go:29] interesting pod/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:00:52.389232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:52.383469    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x" podUID="56528ce3-95f8-408d-96db-04ac7bcab926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 03 20:00:53.401275 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:53.399082    5591 patch_prober.go:29] interesting pod/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 03 20:00:53.401275 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:53.399150    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x" podUID="56528ce3-95f8-408d-96db-04ac7bcab926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 03 20:00:54.381590 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:54.381559    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:54.382771 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:54.382748    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:00:57.478668 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:57.478632    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"]
Nov 03 20:00:57.479453 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:57.479398    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj" podUID="bec03382-01ff-4d4d-89a6-2988ddb9e188" containerName="istio-proxy" containerID="cri-o://7364cc3f424c4fee6e9f69edccc249f0788728588bc80f11f82f307656b00bf1" gracePeriod=30
Nov 03 20:00:58.725756 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:00:58.725719    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:00:58.912112 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:00:58.912071    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:01:00.836379 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:00.836339    5591 patch_prober.go:29] interesting pod/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:01:00.837002 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:00.836968    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj" podUID="bec03382-01ff-4d4d-89a6-2988ddb9e188" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:01:02.299760 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:02.299716    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:02.300686 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:02.300655    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:01:02.368372 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:02.368318    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:02.424247 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:02.424181    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:02.472553 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:02.471337    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:03.023144 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.023097    5591 generic.go:356] "Generic (PLEG): container finished" podID="bec03382-01ff-4d4d-89a6-2988ddb9e188" containerID="7364cc3f424c4fee6e9f69edccc249f0788728588bc80f11f82f307656b00bf1" exitCode=0
Nov 03 20:01:03.023405 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.023160    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj" event={"ID":"bec03382-01ff-4d4d-89a6-2988ddb9e188","Type":"ContainerDied","Data":"7364cc3f424c4fee6e9f69edccc249f0788728588bc80f11f82f307656b00bf1"}
Nov 03 20:01:03.080341 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.080309    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 20:01:03.122750 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.122659    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-podinfo\") pod \"bec03382-01ff-4d4d-89a6-2988ddb9e188\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") "
Nov 03 20:01:03.123003 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.122762    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-envoy\") pod \"bec03382-01ff-4d4d-89a6-2988ddb9e188\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") "
Nov 03 20:01:03.123003 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.122821    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bec03382-01ff-4d4d-89a6-2988ddb9e188-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec03382-01ff-4d4d-89a6-2988ddb9e188-istiod-ca-cert\") pod \"bec03382-01ff-4d4d-89a6-2988ddb9e188\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") "
Nov 03 20:01:03.123145 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.123053    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-workload-socket\") pod \"bec03382-01ff-4d4d-89a6-2988ddb9e188\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") "
Nov 03 20:01:03.123240 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.123164    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bec03382-01ff-4d4d-89a6-2988ddb9e188-kube-api-access-w85xr\" (UniqueName: \"kubernetes.io/projected/bec03382-01ff-4d4d-89a6-2988ddb9e188-kube-api-access-w85xr\") pod \"bec03382-01ff-4d4d-89a6-2988ddb9e188\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") "
Nov 03 20:01:03.123297 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.123254    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-workload-certs\") pod \"bec03382-01ff-4d4d-89a6-2988ddb9e188\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") "
Nov 03 20:01:03.123361 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.123315    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-credential-socket\") pod \"bec03382-01ff-4d4d-89a6-2988ddb9e188\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") "
Nov 03 20:01:03.123361 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.123354    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-token\" (UniqueName: \"kubernetes.io/projected/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-token\") pod \"bec03382-01ff-4d4d-89a6-2988ddb9e188\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") "
Nov 03 20:01:03.123463 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.123389    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-data\") pod \"bec03382-01ff-4d4d-89a6-2988ddb9e188\" (UID: \"bec03382-01ff-4d4d-89a6-2988ddb9e188\") "
Nov 03 20:01:03.127680 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.123814    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-workload-certs" pod "bec03382-01ff-4d4d-89a6-2988ddb9e188" (UID: "bec03382-01ff-4d4d-89a6-2988ddb9e188"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:03.127680 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.124052    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-workload-socket" pod "bec03382-01ff-4d4d-89a6-2988ddb9e188" (UID: "bec03382-01ff-4d4d-89a6-2988ddb9e188"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:03.127680 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.124085    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-data" pod "bec03382-01ff-4d4d-89a6-2988ddb9e188" (UID: "bec03382-01ff-4d4d-89a6-2988ddb9e188"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:03.127680 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.124320    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-credential-socket" pod "bec03382-01ff-4d4d-89a6-2988ddb9e188" (UID: "bec03382-01ff-4d4d-89a6-2988ddb9e188"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:03.127680 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.126707    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bec03382-01ff-4d4d-89a6-2988ddb9e188-istiod-ca-cert" pod "bec03382-01ff-4d4d-89a6-2988ddb9e188" (UID: "bec03382-01ff-4d4d-89a6-2988ddb9e188"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:01:03.128664 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.128631    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:03.129039 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.129010    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-envoy" pod "bec03382-01ff-4d4d-89a6-2988ddb9e188" (UID: "bec03382-01ff-4d4d-89a6-2988ddb9e188"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:03.129435 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.129406    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bec03382-01ff-4d4d-89a6-2988ddb9e188-kube-api-access-w85xr" pod "bec03382-01ff-4d4d-89a6-2988ddb9e188" (UID: "bec03382-01ff-4d4d-89a6-2988ddb9e188"). InnerVolumeSpecName "kube-api-access-w85xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:03.131539 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.131477    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-token" pod "bec03382-01ff-4d4d-89a6-2988ddb9e188" (UID: "bec03382-01ff-4d4d-89a6-2988ddb9e188"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:03.131739 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.131712    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-podinfo" pod "bec03382-01ff-4d4d-89a6-2988ddb9e188" (UID: "bec03382-01ff-4d4d-89a6-2988ddb9e188"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:01:03.229714 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.229675    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w85xr\" (UniqueName: \"kubernetes.io/projected/bec03382-01ff-4d4d-89a6-2988ddb9e188-kube-api-access-w85xr\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:03.229714 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.229709    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:03.229714 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.229719    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:03.229955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.229727    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:03.229955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.229735    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:03.229955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.229746    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:03.229955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.229754    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bec03382-01ff-4d4d-89a6-2988ddb9e188-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:03.229955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:03.229762    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bec03382-01ff-4d4d-89a6-2988ddb9e188-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:04.082477 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:04.082437    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj" event={"ID":"bec03382-01ff-4d4d-89a6-2988ddb9e188","Type":"ContainerDied","Data":"93f9eb320edd1f05c4cf9a48618730ea655342f5ee7c57c8cab0118371284789"}
Nov 03 20:01:04.083085 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:04.082505    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"
Nov 03 20:01:04.083085 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:04.082530    5591 scope.go:122] "RemoveContainer" containerID="7364cc3f424c4fee6e9f69edccc249f0788728588bc80f11f82f307656b00bf1"
Nov 03 20:01:04.104717 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:04.104683    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"]
Nov 03 20:01:04.116534 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:04.116504    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-3d1db1-openshift-default-f56dcd7f9-86crj"]
Nov 03 20:01:05.350724 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:05.350690    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bec03382-01ff-4d4d-89a6-2988ddb9e188" path="/var/lib/kubelet/pods/bec03382-01ff-4d4d-89a6-2988ddb9e188/volumes"
Nov 03 20:01:07.723729 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:07.723695    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f"]
Nov 03 20:01:07.724480 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:07.724431    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f" podUID="4035d2b1-0439-40e5-a01a-209194d91356" containerName="maas-api" containerID="cri-o://f2ed0f056de49d02807bdde29cfe7e6a26e9e16a70fd3598760974f411398e36" gracePeriod=30
Nov 03 20:01:07.806262 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:07.806226    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55"]
Nov 03 20:01:07.806709 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:07.806680    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55" podUID="be582c31-becf-4674-889c-b25de0634f36" containerName="payload-pre-processing" containerID="cri-o://7d4e2d119f4d92326b56ae2654dc377f889e1976efa63e98485684e0c19a2cc0" gracePeriod=30
Nov 03 20:01:08.256396 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:08.255277    5591 generic.go:356] "Generic (PLEG): container finished" podID="be582c31-becf-4674-889c-b25de0634f36" containerID="7d4e2d119f4d92326b56ae2654dc377f889e1976efa63e98485684e0c19a2cc0" exitCode=0
Nov 03 20:01:08.256396 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:08.255421    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55" event={"ID":"be582c31-becf-4674-889c-b25de0634f36","Type":"ContainerDied","Data":"7d4e2d119f4d92326b56ae2654dc377f889e1976efa63e98485684e0c19a2cc0"}
Nov 03 20:01:08.257677 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:08.257005    5591 generic.go:356] "Generic (PLEG): container finished" podID="4035d2b1-0439-40e5-a01a-209194d91356" containerID="f2ed0f056de49d02807bdde29cfe7e6a26e9e16a70fd3598760974f411398e36" exitCode=0
Nov 03 20:01:08.257677 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:08.257102    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f" event={"ID":"4035d2b1-0439-40e5-a01a-209194d91356","Type":"ContainerDied","Data":"f2ed0f056de49d02807bdde29cfe7e6a26e9e16a70fd3598760974f411398e36"}
Nov 03 20:01:08.309904 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:08.309541    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55"
Nov 03 20:01:08.383044 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:08.383015    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f"
Nov 03 20:01:08.511658 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:08.511628    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be582c31-becf-4674-889c-b25de0634f36-kube-api-access-9frvl\" (UniqueName: \"kubernetes.io/projected/be582c31-becf-4674-889c-b25de0634f36-kube-api-access-9frvl\") pod \"be582c31-becf-4674-889c-b25de0634f36\" (UID: \"be582c31-becf-4674-889c-b25de0634f36\") "
Nov 03 20:01:08.511845 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:08.511742    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4035d2b1-0439-40e5-a01a-209194d91356-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4035d2b1-0439-40e5-a01a-209194d91356-openshift-service-ca\") pod \"4035d2b1-0439-40e5-a01a-209194d91356\" (UID: \"4035d2b1-0439-40e5-a01a-209194d91356\") "
Nov 03 20:01:08.511845 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:08.511772    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4035d2b1-0439-40e5-a01a-209194d91356-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4035d2b1-0439-40e5-a01a-209194d91356-metrics-certs\") pod \"4035d2b1-0439-40e5-a01a-209194d91356\" (UID: \"4035d2b1-0439-40e5-a01a-209194d91356\") "
Nov 03 20:01:08.511845 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:08.511822    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4035d2b1-0439-40e5-a01a-209194d91356-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4035d2b1-0439-40e5-a01a-209194d91356-maas-api-tls\") pod \"4035d2b1-0439-40e5-a01a-209194d91356\" (UID: \"4035d2b1-0439-40e5-a01a-209194d91356\") "
Nov 03 20:01:08.512046 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:08.511880    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be582c31-becf-4674-889c-b25de0634f36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be582c31-becf-4674-889c-b25de0634f36-plugins-config-volume\") pod \"be582c31-becf-4674-889c-b25de0634f36\" (UID: \"be582c31-becf-4674-889c-b25de0634f36\") "
Nov 03 20:01:08.512046 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:08.511952    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4035d2b1-0439-40e5-a01a-209194d91356-kube-api-access-f49jr\" (UniqueName: \"kubernetes.io/projected/4035d2b1-0439-40e5-a01a-209194d91356-kube-api-access-f49jr\") pod \"4035d2b1-0439-40e5-a01a-209194d91356\" (UID: \"4035d2b1-0439-40e5-a01a-209194d91356\") "
Nov 03 20:01:08.512529 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:08.512494    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be582c31-becf-4674-889c-b25de0634f36-plugins-config-volume" pod "be582c31-becf-4674-889c-b25de0634f36" (UID: "be582c31-becf-4674-889c-b25de0634f36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:01:08.512871 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:08.512835    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4035d2b1-0439-40e5-a01a-209194d91356-openshift-service-ca" pod "4035d2b1-0439-40e5-a01a-209194d91356" (UID: "4035d2b1-0439-40e5-a01a-209194d91356"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:01:08.513606 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:08.513583    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4035d2b1-0439-40e5-a01a-209194d91356-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:08.515136 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:08.515121    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be582c31-becf-4674-889c-b25de0634f36-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:08.515858 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:08.515823    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be582c31-becf-4674-889c-b25de0634f36-kube-api-access-9frvl" pod "be582c31-becf-4674-889c-b25de0634f36" (UID: "be582c31-becf-4674-889c-b25de0634f36"). InnerVolumeSpecName "kube-api-access-9frvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:08.515968 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:08.515885    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4035d2b1-0439-40e5-a01a-209194d91356-metrics-certs" pod "4035d2b1-0439-40e5-a01a-209194d91356" (UID: "4035d2b1-0439-40e5-a01a-209194d91356"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:01:08.516471 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:08.516406    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4035d2b1-0439-40e5-a01a-209194d91356-maas-api-tls" pod "4035d2b1-0439-40e5-a01a-209194d91356" (UID: "4035d2b1-0439-40e5-a01a-209194d91356"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:01:08.517918 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:08.517894    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4035d2b1-0439-40e5-a01a-209194d91356-kube-api-access-f49jr" pod "4035d2b1-0439-40e5-a01a-209194d91356" (UID: "4035d2b1-0439-40e5-a01a-209194d91356"). InnerVolumeSpecName "kube-api-access-f49jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:08.617015 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:08.616934    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f49jr\" (UniqueName: \"kubernetes.io/projected/4035d2b1-0439-40e5-a01a-209194d91356-kube-api-access-f49jr\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:08.617015 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:08.616963    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9frvl\" (UniqueName: \"kubernetes.io/projected/be582c31-becf-4674-889c-b25de0634f36-kube-api-access-9frvl\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:08.617015 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:08.616973    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4035d2b1-0439-40e5-a01a-209194d91356-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:08.617015 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:08.616982    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4035d2b1-0439-40e5-a01a-209194d91356-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:09.265493 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:09.265464    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55"
Nov 03 20:01:09.265493 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:09.265484    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55" event={"ID":"be582c31-becf-4674-889c-b25de0634f36","Type":"ContainerDied","Data":"07929d4ae18eb14530e3c2a7907f2c6334d87463c2ba9dfc15cd312db057510d"}
Nov 03 20:01:09.266070 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:09.265583    5591 scope.go:122] "RemoveContainer" containerID="7d4e2d119f4d92326b56ae2654dc377f889e1976efa63e98485684e0c19a2cc0"
Nov 03 20:01:09.267619 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:09.267591    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f" event={"ID":"4035d2b1-0439-40e5-a01a-209194d91356","Type":"ContainerDied","Data":"ba1fb2ea352c28a5e23829b813f3cd7a493aa3446c2b104040bcf65f559535c9"}
Nov 03 20:01:09.267619 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:09.267609    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f"
Nov 03 20:01:09.277445 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:09.277156    5591 scope.go:122] "RemoveContainer" containerID="f2ed0f056de49d02807bdde29cfe7e6a26e9e16a70fd3598760974f411398e36"
Nov 03 20:01:09.295707 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:09.295680    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55"]
Nov 03 20:01:09.300374 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:09.300344    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c6092e-74cbff7f67-kqh55"]
Nov 03 20:01:09.308031 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:09.307997    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f"]
Nov 03 20:01:09.311094 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:09.311072    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c6092e-6f5968998d-6sb6f"]
Nov 03 20:01:09.350272 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:09.350239    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4035d2b1-0439-40e5-a01a-209194d91356" path="/var/lib/kubelet/pods/4035d2b1-0439-40e5-a01a-209194d91356/volumes"
Nov 03 20:01:09.350698 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:09.350685    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be582c31-becf-4674-889c-b25de0634f36" path="/var/lib/kubelet/pods/be582c31-becf-4674-889c-b25de0634f36/volumes"
Nov 03 20:01:21.148711 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:21.148676    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2"]
Nov 03 20:01:23.993874 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:23.993834    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:01:26.157232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:26.151287    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2" podUID="836a78aa-c8d3-47c2-8b42-3bfd919fa477" containerName="payload-pre-processing" containerID="cri-o://5ec71ebb619cbe952e9d9ca2c58e395ad74d10b9a157b2a1480f77f7167cad86" gracePeriod=25
Nov 03 20:01:26.341470 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:26.341413    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:26.452473 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:26.452370    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:26.453500 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:26.453471    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 20:01:26.481342 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:26.481293    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:01:26.895230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:26.894311    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:27.045368 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:27.045307    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:27.346363 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:27.343312    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:27.346363 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:27.343793    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 20:01:27.379884 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:27.372367    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:27.476583 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:27.475690    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:27.476583 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:27.476095    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 20:01:27.507496 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:27.507456    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:27.748726 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:27.748628    5591 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:01:27.749017 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:27.748983    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm" podUID="e31559fd-2ec2-48cd-8f00-8167a567097b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:28.139714 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:27.985525    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:01:28.244232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:28.215815    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:28.296639 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:28.284555    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:28.376815 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:28.376481    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:28.435358 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:28.434359    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:28.502282 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:28.500301    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 20:01:28.502282 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:28.501651    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 20:01:28.545317 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:28.545273    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:01:28.928230 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:01:28.792037    5591 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.401s"
Nov 03 20:01:28.928230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:28.831686    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 20:01:28.928230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:28.831738    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 20:01:28.928230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:28.831770    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 20:01:28.928230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:28.831806    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 20:01:28.928230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:28.831842    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 20:01:28.928230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:28.831873    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 20:01:28.928230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:28.831902    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 20:01:28.928230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:28.831931    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 20:01:28.928230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:28.831957    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 20:01:28.928230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:28.835707    5591 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:01:28.928230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:28.835781    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7" podUID="3c70d023-1700-42d0-964f-5573383d1355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:28.990031 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:28.989978    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:29.060816 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:29.060415    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 20:01:29.102664 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:29.099672    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 20:01:29.134649 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:29.134619    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 20:01:29.150246 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:29.150219    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 20:01:29.228182 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:01:29.228081    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:01:30.801738 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.801708    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"]
Nov 03 20:01:30.802388 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802374    5591 cpu_manager.go:414] "containerMap: removing container" podUID="3efe99ef-4e27-4f9c-994b-ebdd23792b8c" containerName="payload-processing"
Nov 03 20:01:30.802444 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802389    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3efe99ef-4e27-4f9c-994b-ebdd23792b8c" containerName="payload-processing"
Nov 03 20:01:30.802444 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802406    5591 cpu_manager.go:414] "containerMap: removing container" podUID="09c9289a-f596-4c46-b4a0-a85cc1cc70e5" containerName="payload-pre-processing"
Nov 03 20:01:30.802444 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802412    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c9289a-f596-4c46-b4a0-a85cc1cc70e5" containerName="payload-pre-processing"
Nov 03 20:01:30.802444 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802421    5591 cpu_manager.go:414] "containerMap: removing container" podUID="4dc9dccb-784a-4082-9a7f-39223a74ce3b" containerName="payload-processing"
Nov 03 20:01:30.802444 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802426    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dc9dccb-784a-4082-9a7f-39223a74ce3b" containerName="payload-processing"
Nov 03 20:01:30.802444 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802441    5591 cpu_manager.go:414] "containerMap: removing container" podUID="a08831b1-e194-4e39-b23d-6232d8d9851e" containerName="istio-proxy"
Nov 03 20:01:30.802444 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802447    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a08831b1-e194-4e39-b23d-6232d8d9851e" containerName="istio-proxy"
Nov 03 20:01:30.802683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802455    5591 cpu_manager.go:414] "containerMap: removing container" podUID="43ad69e4-a321-4324-9e2a-9789fed758e9" containerName="payload-pre-processing"
Nov 03 20:01:30.802683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802459    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ad69e4-a321-4324-9e2a-9789fed758e9" containerName="payload-pre-processing"
Nov 03 20:01:30.802683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802469    5591 cpu_manager.go:414] "containerMap: removing container" podUID="12d2b7e4-96b2-4ee9-a511-4ca0f21228fe" containerName="maas-api"
Nov 03 20:01:30.802683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802473    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12d2b7e4-96b2-4ee9-a511-4ca0f21228fe" containerName="maas-api"
Nov 03 20:01:30.802683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802481    5591 cpu_manager.go:414] "containerMap: removing container" podUID="be582c31-becf-4674-889c-b25de0634f36" containerName="payload-pre-processing"
Nov 03 20:01:30.802683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802485    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be582c31-becf-4674-889c-b25de0634f36" containerName="payload-pre-processing"
Nov 03 20:01:30.802683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802493    5591 cpu_manager.go:414] "containerMap: removing container" podUID="928410e8-ddb8-4a61-8a41-f218603c0e47" containerName="istio-proxy"
Nov 03 20:01:30.802683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802497    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="928410e8-ddb8-4a61-8a41-f218603c0e47" containerName="istio-proxy"
Nov 03 20:01:30.802683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802504    5591 cpu_manager.go:414] "containerMap: removing container" podUID="bec03382-01ff-4d4d-89a6-2988ddb9e188" containerName="istio-proxy"
Nov 03 20:01:30.802683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802508    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bec03382-01ff-4d4d-89a6-2988ddb9e188" containerName="istio-proxy"
Nov 03 20:01:30.802683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802513    5591 cpu_manager.go:414] "containerMap: removing container" podUID="4035d2b1-0439-40e5-a01a-209194d91356" containerName="maas-api"
Nov 03 20:01:30.802683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802517    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4035d2b1-0439-40e5-a01a-209194d91356" containerName="maas-api"
Nov 03 20:01:30.802683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802526    5591 cpu_manager.go:414] "containerMap: removing container" podUID="270abfe6-e1dc-4a2e-a106-4cd071d79612" containerName="payload-pre-processing"
Nov 03 20:01:30.802683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802531    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="270abfe6-e1dc-4a2e-a106-4cd071d79612" containerName="payload-pre-processing"
Nov 03 20:01:30.802683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802538    5591 cpu_manager.go:414] "containerMap: removing container" podUID="ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f" containerName="payload-pre-processing"
Nov 03 20:01:30.802683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802542    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f" containerName="payload-pre-processing"
Nov 03 20:01:30.802683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802550    5591 cpu_manager.go:414] "containerMap: removing container" podUID="e1591b82-68d7-4343-980b-8d6e1cce458c" containerName="payload-processing"
Nov 03 20:01:30.802683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802554    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1591b82-68d7-4343-980b-8d6e1cce458c" containerName="payload-processing"
Nov 03 20:01:30.802683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802565    5591 cpu_manager.go:414] "containerMap: removing container" podUID="bf807ba9-d308-4b56-96a4-702135b4d522" containerName="maas-api"
Nov 03 20:01:30.802683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802569    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf807ba9-d308-4b56-96a4-702135b4d522" containerName="maas-api"
Nov 03 20:01:30.802683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802652    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="a08831b1-e194-4e39-b23d-6232d8d9851e" containerName="istio-proxy"
Nov 03 20:01:30.802683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802662    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ad69e4-a321-4324-9e2a-9789fed758e9" containerName="payload-pre-processing"
Nov 03 20:01:30.802683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802668    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="12d2b7e4-96b2-4ee9-a511-4ca0f21228fe" containerName="maas-api"
Nov 03 20:01:30.802683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802678    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="be582c31-becf-4674-889c-b25de0634f36" containerName="payload-pre-processing"
Nov 03 20:01:30.802683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802684    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="928410e8-ddb8-4a61-8a41-f218603c0e47" containerName="istio-proxy"
Nov 03 20:01:30.802683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802691    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="bec03382-01ff-4d4d-89a6-2988ddb9e188" containerName="istio-proxy"
Nov 03 20:01:30.803482 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802696    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="4035d2b1-0439-40e5-a01a-209194d91356" containerName="maas-api"
Nov 03 20:01:30.803482 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802702    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="270abfe6-e1dc-4a2e-a106-4cd071d79612" containerName="payload-pre-processing"
Nov 03 20:01:30.803482 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802706    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddb21a4b-a11b-4ff9-89d5-2e2edc982b9f" containerName="payload-pre-processing"
Nov 03 20:01:30.803482 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802712    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1591b82-68d7-4343-980b-8d6e1cce458c" containerName="payload-processing"
Nov 03 20:01:30.803482 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802718    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf807ba9-d308-4b56-96a4-702135b4d522" containerName="maas-api"
Nov 03 20:01:30.803482 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802725    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3efe99ef-4e27-4f9c-994b-ebdd23792b8c" containerName="payload-processing"
Nov 03 20:01:30.803482 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802732    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c9289a-f596-4c46-b4a0-a85cc1cc70e5" containerName="payload-pre-processing"
Nov 03 20:01:30.803482 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.802737    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dc9dccb-784a-4082-9a7f-39223a74ce3b" containerName="payload-processing"
Nov 03 20:01:30.806640 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.806615    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:30.823979 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.823924    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"]
Nov 03 20:01:30.866199 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.866168    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:30.866426 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.866362    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa169e55-a457-4166-88ab-c65b6081d08e-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:30.866426 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.866419    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:30.866584 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.866562    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:30.866642 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.866605    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa169e55-a457-4166-88ab-c65b6081d08e-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:30.866706 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.866692    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa169e55-a457-4166-88ab-c65b6081d08e-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:30.866747 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.866717    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:30.866796 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.866758    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:30.866863 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.866843    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt89x\" (UniqueName: \"kubernetes.io/projected/fa169e55-a457-4166-88ab-c65b6081d08e-kube-api-access-nt89x\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:30.968948 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.968908    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:30.969358 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.969333    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt89x\" (UniqueName: \"kubernetes.io/projected/fa169e55-a457-4166-88ab-c65b6081d08e-kube-api-access-nt89x\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:30.969610 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.969591    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:30.969881 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.969811    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa169e55-a457-4166-88ab-c65b6081d08e-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:30.970056 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.970038    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:30.970301 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.970282    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:30.970966 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.970519    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa169e55-a457-4166-88ab-c65b6081d08e-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:30.970966 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.970527    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa169e55-a457-4166-88ab-c65b6081d08e-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:30.970966 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.970629    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa169e55-a457-4166-88ab-c65b6081d08e-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:30.970966 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.970663    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:30.971256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.969845    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:30.971256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.971043    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:30.971437 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.971413    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:30.971564 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.969610    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:30.974372 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.974347    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:30.974587 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.974569    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa169e55-a457-4166-88ab-c65b6081d08e-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:30.989327 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:30.989263    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt89x\" (UniqueName: \"kubernetes.io/projected/fa169e55-a457-4166-88ab-c65b6081d08e-kube-api-access-nt89x\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:31.009041 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:31.009007    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa169e55-a457-4166-88ab-c65b6081d08e-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:31.049125 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:31.048830    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2" podUID="836a78aa-c8d3-47c2-8b42-3bfd919fa477" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:01:31.130104 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:31.130000    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:31.420751 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:31.420723    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"]
Nov 03 20:01:31.421690 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:01:31.421664    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa169e55_a457_4166_88ab_c65b6081d08e.slice/crio-a67fd9a03d27d0403fd76fcb693b1cd6d4dc3cd8670f06a8b40da426f03af23a WatchSource:0}: Error finding container a67fd9a03d27d0403fd76fcb693b1cd6d4dc3cd8670f06a8b40da426f03af23a: Status 404 returned error can't find the container with id a67fd9a03d27d0403fd76fcb693b1cd6d4dc3cd8670f06a8b40da426f03af23a
Nov 03 20:01:31.424368 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:31.424328    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:01:31.424460 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:31.424397    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:01:31.424460 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:31.424428    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:01:31.796925 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:31.796888    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb" event={"ID":"fa169e55-a457-4166-88ab-c65b6081d08e","Type":"ContainerStarted","Data":"933b9e58dc85e048a725a99718543b410c566438a639d79df807c099d4cea182"}
Nov 03 20:01:31.797140 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:31.796948    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb" event={"ID":"fa169e55-a457-4166-88ab-c65b6081d08e","Type":"ContainerStarted","Data":"a67fd9a03d27d0403fd76fcb693b1cd6d4dc3cd8670f06a8b40da426f03af23a"}
Nov 03 20:01:31.822484 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:31.822435    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb" podStartSLOduration=1.822421373 podStartE2EDuration="1.822421373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:31.820115104 +0000 UTC m=+1440.592408122" watchObservedRunningTime="2026-11-03 20:01:31.822421373 +0000 UTC m=+1440.594714390"
Nov 03 20:01:32.130819 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:32.130735    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:32.132861 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:32.132741    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 03 20:01:32.133001 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:32.132888    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb" podUID="fa169e55-a457-4166-88ab-c65b6081d08e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 03 20:01:33.130618 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:33.130576    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 03 20:01:33.131119 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:33.130638    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb" podUID="fa169e55-a457-4166-88ab-c65b6081d08e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 03 20:01:34.131125 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:34.131087    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 03 20:01:34.131617 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:34.131148    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb" podUID="fa169e55-a457-4166-88ab-c65b6081d08e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 03 20:01:34.326430 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:34.326093    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj"]
Nov 03 20:01:34.326700 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:34.326390    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj" podUID="d141244a-fd87-4e3c-90aa-f4b1cf60aa84" containerName="maas-api" containerID="cri-o://0f160857af46695d84c272391556ffec300a7d1bb9b5a8068b50493645d2615f" gracePeriod=30
Nov 03 20:01:34.929638 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:34.929566    5591 generic.go:356] "Generic (PLEG): container finished" podID="d141244a-fd87-4e3c-90aa-f4b1cf60aa84" containerID="0f160857af46695d84c272391556ffec300a7d1bb9b5a8068b50493645d2615f" exitCode=0
Nov 03 20:01:34.929829 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:34.929794    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj" event={"ID":"d141244a-fd87-4e3c-90aa-f4b1cf60aa84","Type":"ContainerDied","Data":"0f160857af46695d84c272391556ffec300a7d1bb9b5a8068b50493645d2615f"}
Nov 03 20:01:34.952985 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:34.952960    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj"
Nov 03 20:01:35.053574 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:35.053545    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-kube-api-access-kdtlp\" (UniqueName: \"kubernetes.io/projected/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-kube-api-access-kdtlp\") pod \"d141244a-fd87-4e3c-90aa-f4b1cf60aa84\" (UID: \"d141244a-fd87-4e3c-90aa-f4b1cf60aa84\") "
Nov 03 20:01:35.053774 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:35.053660    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-openshift-service-ca\") pod \"d141244a-fd87-4e3c-90aa-f4b1cf60aa84\" (UID: \"d141244a-fd87-4e3c-90aa-f4b1cf60aa84\") "
Nov 03 20:01:35.053774 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:35.053761    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-metrics-certs\") pod \"d141244a-fd87-4e3c-90aa-f4b1cf60aa84\" (UID: \"d141244a-fd87-4e3c-90aa-f4b1cf60aa84\") "
Nov 03 20:01:35.054030 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:35.053992    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-openshift-service-ca" pod "d141244a-fd87-4e3c-90aa-f4b1cf60aa84" (UID: "d141244a-fd87-4e3c-90aa-f4b1cf60aa84"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:01:35.054160 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:35.054142    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-maas-api-tls\") pod \"d141244a-fd87-4e3c-90aa-f4b1cf60aa84\" (UID: \"d141244a-fd87-4e3c-90aa-f4b1cf60aa84\") "
Nov 03 20:01:35.055280 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:35.055264    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:35.056184 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:35.056154    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-kube-api-access-kdtlp" pod "d141244a-fd87-4e3c-90aa-f4b1cf60aa84" (UID: "d141244a-fd87-4e3c-90aa-f4b1cf60aa84"). InnerVolumeSpecName "kube-api-access-kdtlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:35.056439 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:35.056413    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-metrics-certs" pod "d141244a-fd87-4e3c-90aa-f4b1cf60aa84" (UID: "d141244a-fd87-4e3c-90aa-f4b1cf60aa84"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:01:35.057352 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:35.057337    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-maas-api-tls" pod "d141244a-fd87-4e3c-90aa-f4b1cf60aa84" (UID: "d141244a-fd87-4e3c-90aa-f4b1cf60aa84"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:01:35.168316 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:35.164659    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdtlp\" (UniqueName: \"kubernetes.io/projected/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-kube-api-access-kdtlp\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:35.168316 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:35.164696    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:35.168316 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:35.164713    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d141244a-fd87-4e3c-90aa-f4b1cf60aa84-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:35.168316 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:35.165141    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 03 20:01:35.193271 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:35.165192    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb" podUID="fa169e55-a457-4166-88ab-c65b6081d08e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 03 20:01:35.869307 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:35.869271    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w"]
Nov 03 20:01:35.869837 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:35.869805    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w" podUID="7a6377f5-2c6f-4005-bac4-9caba84e3314" containerName="maas-api" containerID="cri-o://1642f1354002650aacbb3ae1237a1364d0ffe77e7944a0d268638b662e5767db" gracePeriod=30
Nov 03 20:01:35.939970 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:35.939939    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj"
Nov 03 20:01:35.940356 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:35.940327    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj" event={"ID":"d141244a-fd87-4e3c-90aa-f4b1cf60aa84","Type":"ContainerDied","Data":"9a293e04ab614c6875184991343586b81bea05691af47d56bcb62ef3dbe14a22"}
Nov 03 20:01:35.940514 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:35.940500    5591 scope.go:122] "RemoveContainer" containerID="0f160857af46695d84c272391556ffec300a7d1bb9b5a8068b50493645d2615f"
Nov 03 20:01:35.964373 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:35.964342    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj"]
Nov 03 20:01:35.969229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:35.969183    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8b3020-68695fc48c-42xpj"]
Nov 03 20:01:36.139832 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:36.139727    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 03 20:01:36.140016 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:36.139850    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb" podUID="fa169e55-a457-4166-88ab-c65b6081d08e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 03 20:01:36.659312 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:36.659281    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w"
Nov 03 20:01:36.712797 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:36.712271    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a6377f5-2c6f-4005-bac4-9caba84e3314-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a6377f5-2c6f-4005-bac4-9caba84e3314-metrics-certs\") pod \"7a6377f5-2c6f-4005-bac4-9caba84e3314\" (UID: \"7a6377f5-2c6f-4005-bac4-9caba84e3314\") "
Nov 03 20:01:36.712797 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:36.712451    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a6377f5-2c6f-4005-bac4-9caba84e3314-kube-api-access-mtgmp\" (UniqueName: \"kubernetes.io/projected/7a6377f5-2c6f-4005-bac4-9caba84e3314-kube-api-access-mtgmp\") pod \"7a6377f5-2c6f-4005-bac4-9caba84e3314\" (UID: \"7a6377f5-2c6f-4005-bac4-9caba84e3314\") "
Nov 03 20:01:36.712797 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:36.712497    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a6377f5-2c6f-4005-bac4-9caba84e3314-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6377f5-2c6f-4005-bac4-9caba84e3314-openshift-service-ca\") pod \"7a6377f5-2c6f-4005-bac4-9caba84e3314\" (UID: \"7a6377f5-2c6f-4005-bac4-9caba84e3314\") "
Nov 03 20:01:36.713091 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:36.713033    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a6377f5-2c6f-4005-bac4-9caba84e3314-openshift-service-ca" pod "7a6377f5-2c6f-4005-bac4-9caba84e3314" (UID: "7a6377f5-2c6f-4005-bac4-9caba84e3314"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:01:36.713193 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:36.713154    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a6377f5-2c6f-4005-bac4-9caba84e3314-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a6377f5-2c6f-4005-bac4-9caba84e3314-maas-api-tls\") pod \"7a6377f5-2c6f-4005-bac4-9caba84e3314\" (UID: \"7a6377f5-2c6f-4005-bac4-9caba84e3314\") "
Nov 03 20:01:36.714194 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:36.714166    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a6377f5-2c6f-4005-bac4-9caba84e3314-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:36.715369 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:36.715341    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a6377f5-2c6f-4005-bac4-9caba84e3314-kube-api-access-mtgmp" pod "7a6377f5-2c6f-4005-bac4-9caba84e3314" (UID: "7a6377f5-2c6f-4005-bac4-9caba84e3314"). InnerVolumeSpecName "kube-api-access-mtgmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:36.715952 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:36.715916    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a6377f5-2c6f-4005-bac4-9caba84e3314-metrics-certs" pod "7a6377f5-2c6f-4005-bac4-9caba84e3314" (UID: "7a6377f5-2c6f-4005-bac4-9caba84e3314"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:01:36.717080 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:36.717054    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a6377f5-2c6f-4005-bac4-9caba84e3314-maas-api-tls" pod "7a6377f5-2c6f-4005-bac4-9caba84e3314" (UID: "7a6377f5-2c6f-4005-bac4-9caba84e3314"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:01:36.816130 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:36.816096    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a6377f5-2c6f-4005-bac4-9caba84e3314-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:36.816130 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:36.816127    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtgmp\" (UniqueName: \"kubernetes.io/projected/7a6377f5-2c6f-4005-bac4-9caba84e3314-kube-api-access-mtgmp\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:36.816130 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:36.816139    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a6377f5-2c6f-4005-bac4-9caba84e3314-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:36.957979 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:36.957943    5591 generic.go:356] "Generic (PLEG): container finished" podID="7a6377f5-2c6f-4005-bac4-9caba84e3314" containerID="1642f1354002650aacbb3ae1237a1364d0ffe77e7944a0d268638b662e5767db" exitCode=0
Nov 03 20:01:36.958194 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:36.958010    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w"
Nov 03 20:01:36.958194 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:36.958016    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w" event={"ID":"7a6377f5-2c6f-4005-bac4-9caba84e3314","Type":"ContainerDied","Data":"1642f1354002650aacbb3ae1237a1364d0ffe77e7944a0d268638b662e5767db"}
Nov 03 20:01:36.958194 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:36.958110    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w" event={"ID":"7a6377f5-2c6f-4005-bac4-9caba84e3314","Type":"ContainerDied","Data":"3b48a4d6be3fc893ef9721bf659323956e5f64fc54edf33030603d4c0ab12feb"}
Nov 03 20:01:36.958194 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:36.958141    5591 scope.go:122] "RemoveContainer" containerID="1642f1354002650aacbb3ae1237a1364d0ffe77e7944a0d268638b662e5767db"
Nov 03 20:01:36.969191 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:36.969173    5591 scope.go:122] "RemoveContainer" containerID="1642f1354002650aacbb3ae1237a1364d0ffe77e7944a0d268638b662e5767db"
Nov 03 20:01:36.969509 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:01:36.969490    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1642f1354002650aacbb3ae1237a1364d0ffe77e7944a0d268638b662e5767db\": container with ID starting with 1642f1354002650aacbb3ae1237a1364d0ffe77e7944a0d268638b662e5767db not found: ID does not exist" containerID="1642f1354002650aacbb3ae1237a1364d0ffe77e7944a0d268638b662e5767db"
Nov 03 20:01:36.969555 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:36.969518    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1642f1354002650aacbb3ae1237a1364d0ffe77e7944a0d268638b662e5767db"} err="failed to get container status \"1642f1354002650aacbb3ae1237a1364d0ffe77e7944a0d268638b662e5767db\": rpc error: code = NotFound desc = could not find container \"1642f1354002650aacbb3ae1237a1364d0ffe77e7944a0d268638b662e5767db\": container with ID starting with 1642f1354002650aacbb3ae1237a1364d0ffe77e7944a0d268638b662e5767db not found: ID does not exist"
Nov 03 20:01:37.007336 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:37.006278    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w"]
Nov 03 20:01:37.009827 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:37.009786    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-1b1d04-6b86cdb669-z288w"]
Nov 03 20:01:37.130563 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:37.130531    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 03 20:01:37.130733 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:37.130598    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb" podUID="fa169e55-a457-4166-88ab-c65b6081d08e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 03 20:01:37.350858 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:37.350815    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a6377f5-2c6f-4005-bac4-9caba84e3314" path="/var/lib/kubelet/pods/7a6377f5-2c6f-4005-bac4-9caba84e3314/volumes"
Nov 03 20:01:37.351478 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:37.351453    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d141244a-fd87-4e3c-90aa-f4b1cf60aa84" path="/var/lib/kubelet/pods/d141244a-fd87-4e3c-90aa-f4b1cf60aa84/volumes"
Nov 03 20:01:38.130830 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:38.130789    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Nov 03 20:01:38.131355 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:38.130866    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb" podUID="fa169e55-a457-4166-88ab-c65b6081d08e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Nov 03 20:01:39.785109 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:39.785076    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:39.785596 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:39.785429    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:39.786361 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:39.786339    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:01:40.072909 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:40.072831    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"]
Nov 03 20:01:40.073086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:40.073057    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x" podUID="56528ce3-95f8-408d-96db-04ac7bcab926" containerName="istio-proxy" containerID="cri-o://3b4385c98ae8bb6442e29c7c255251d46546c8adb221f4cacd1450be56f406f5" gracePeriod=30
Nov 03 20:01:41.030964 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:41.030924    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2" podUID="836a78aa-c8d3-47c2-8b42-3bfd919fa477" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:01:42.038288 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.038251    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"]
Nov 03 20:01:42.039048 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.039024    5591 cpu_manager.go:414] "containerMap: removing container" podUID="7a6377f5-2c6f-4005-bac4-9caba84e3314" containerName="maas-api"
Nov 03 20:01:42.039255 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.039236    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a6377f5-2c6f-4005-bac4-9caba84e3314" containerName="maas-api"
Nov 03 20:01:42.039403 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.039390    5591 cpu_manager.go:414] "containerMap: removing container" podUID="d141244a-fd87-4e3c-90aa-f4b1cf60aa84" containerName="maas-api"
Nov 03 20:01:42.039511 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.039491    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d141244a-fd87-4e3c-90aa-f4b1cf60aa84" containerName="maas-api"
Nov 03 20:01:42.039704 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.039689    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="d141244a-fd87-4e3c-90aa-f4b1cf60aa84" containerName="maas-api"
Nov 03 20:01:42.039868 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.039725    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a6377f5-2c6f-4005-bac4-9caba84e3314" containerName="maas-api"
Nov 03 20:01:42.045098 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.045077    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.058804 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.058771    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"]
Nov 03 20:01:42.059243 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.059186    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-8fa615-openshift-default-dockercfg-7zgdw\""
Nov 03 20:01:42.176230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.167088    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-envoy\") pod \"e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") " pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.176230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.167225    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-workload-certs\") pod \"e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") " pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.176230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.167262    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mjmk\" (UniqueName: \"kubernetes.io/projected/b12a508d-cf94-477f-8a7b-856231ac6aa5-kube-api-access-2mjmk\") pod \"e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") " pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.176230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.167311    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-token\") pod \"e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") " pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.176230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.167368    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12a508d-cf94-477f-8a7b-856231ac6aa5-istiod-ca-cert\") pod \"e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") " pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.176230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.167615    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-workload-socket\") pod \"e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") " pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.176230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.167672    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-podinfo\") pod \"e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") " pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.176230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.167718    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-credential-socket\") pod \"e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") " pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.176230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.167755    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-data\") pod \"e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") " pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.281238 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.281180    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-workload-socket\") pod \"e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") " pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.281510 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.281492    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-podinfo\") pod \"e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") " pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.281645 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.281631    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-credential-socket\") pod \"e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") " pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.281757 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.281743    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-data\") pod \"e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") " pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.281898 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.281885    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-envoy\") pod \"e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") " pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.282033 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.282020    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-workload-certs\") pod \"e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") " pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.282143 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.282125    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mjmk\" (UniqueName: \"kubernetes.io/projected/b12a508d-cf94-477f-8a7b-856231ac6aa5-kube-api-access-2mjmk\") pod \"e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") " pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.282295 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.282276    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-token\") pod \"e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") " pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.282431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.282418    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12a508d-cf94-477f-8a7b-856231ac6aa5-istiod-ca-cert\") pod \"e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") " pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.288739 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.283431    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12a508d-cf94-477f-8a7b-856231ac6aa5-istiod-ca-cert\") pod \"e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") " pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.288978 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.283510    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-data\") pod \"e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") " pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.289105 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.283725    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-workload-socket\") pod \"e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") " pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.290830 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.284441    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-credential-socket\") pod \"e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") " pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.291020 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.284665    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-workload-certs\") pod \"e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") " pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.292782 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.290742    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-envoy\") pod \"e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") " pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.306108 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.305490    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-podinfo\") pod \"e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") " pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.326609 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.325856    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-token\") pod \"e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") " pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.331866 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.328707    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mjmk\" (UniqueName: \"kubernetes.io/projected/b12a508d-cf94-477f-8a7b-856231ac6aa5-kube-api-access-2mjmk\") pod \"e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") " pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.375234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.373563    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:42.695987 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.695950    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"]
Nov 03 20:01:42.698570 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:01:42.698537    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb12a508d_cf94_477f_8a7b_856231ac6aa5.slice/crio-9d0ad6bd0d7bf1c8a44d32a16aa8a529bfd0cb0b625716f69ab07a3e41dfd379 WatchSource:0}: Error finding container 9d0ad6bd0d7bf1c8a44d32a16aa8a529bfd0cb0b625716f69ab07a3e41dfd379: Status 404 returned error can't find the container with id 9d0ad6bd0d7bf1c8a44d32a16aa8a529bfd0cb0b625716f69ab07a3e41dfd379
Nov 03 20:01:42.700963 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.700926    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:01:42.701049 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.700996    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:01:42.701049 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:42.701028    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:01:43.035581 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:43.035546    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght" event={"ID":"b12a508d-cf94-477f-8a7b-856231ac6aa5","Type":"ContainerStarted","Data":"431e5b0f265a02c8259b3be9f7f15edfc112cb2c0f3d46655f0b2cfd561be69a"}
Nov 03 20:01:43.035770 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:43.035617    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght" event={"ID":"b12a508d-cf94-477f-8a7b-856231ac6aa5","Type":"ContainerStarted","Data":"9d0ad6bd0d7bf1c8a44d32a16aa8a529bfd0cb0b625716f69ab07a3e41dfd379"}
Nov 03 20:01:43.062189 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:43.062128    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght" podStartSLOduration=1.062111659 podStartE2EDuration="1.062111659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:43.058515541 +0000 UTC m=+1451.830808558" watchObservedRunningTime="2026-11-03 20:01:43.062111659 +0000 UTC m=+1451.834404676"
Nov 03 20:01:43.381263 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:43.381161    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:43.386764 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:43.386741    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:44.043276 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:44.043246    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:44.044190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:44.044174    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:01:45.368063 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.368002    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:01:45.443335 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.443301    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56528ce3-95f8-408d-96db-04ac7bcab926-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56528ce3-95f8-408d-96db-04ac7bcab926-istio-podinfo\") pod \"56528ce3-95f8-408d-96db-04ac7bcab926\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") "
Nov 03 20:01:45.443548 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.443384    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56528ce3-95f8-408d-96db-04ac7bcab926-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56528ce3-95f8-408d-96db-04ac7bcab926-istiod-ca-cert\") pod \"56528ce3-95f8-408d-96db-04ac7bcab926\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") "
Nov 03 20:01:45.443548 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.443509    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-workload-certs\") pod \"56528ce3-95f8-408d-96db-04ac7bcab926\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") "
Nov 03 20:01:45.443790 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.443767    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-workload-certs" pod "56528ce3-95f8-408d-96db-04ac7bcab926" (UID: "56528ce3-95f8-408d-96db-04ac7bcab926"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:45.443847 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.443815    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56528ce3-95f8-408d-96db-04ac7bcab926-istiod-ca-cert" pod "56528ce3-95f8-408d-96db-04ac7bcab926" (UID: "56528ce3-95f8-408d-96db-04ac7bcab926"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:01:45.443900 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.443887    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-istio-envoy\") pod \"56528ce3-95f8-408d-96db-04ac7bcab926\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") "
Nov 03 20:01:45.444010 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.443992    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-istio-data\") pod \"56528ce3-95f8-408d-96db-04ac7bcab926\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") "
Nov 03 20:01:45.444068 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.444033    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-credential-socket\") pod \"56528ce3-95f8-408d-96db-04ac7bcab926\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") "
Nov 03 20:01:45.444118 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.444072    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-workload-socket\") pod \"56528ce3-95f8-408d-96db-04ac7bcab926\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") "
Nov 03 20:01:45.444171 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.444145    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56528ce3-95f8-408d-96db-04ac7bcab926-istio-token\" (UniqueName: \"kubernetes.io/projected/56528ce3-95f8-408d-96db-04ac7bcab926-istio-token\") pod \"56528ce3-95f8-408d-96db-04ac7bcab926\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") "
Nov 03 20:01:45.444262 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.444186    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56528ce3-95f8-408d-96db-04ac7bcab926-kube-api-access-vnhhn\" (UniqueName: \"kubernetes.io/projected/56528ce3-95f8-408d-96db-04ac7bcab926-kube-api-access-vnhhn\") pod \"56528ce3-95f8-408d-96db-04ac7bcab926\" (UID: \"56528ce3-95f8-408d-96db-04ac7bcab926\") "
Nov 03 20:01:45.444957 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.444936    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56528ce3-95f8-408d-96db-04ac7bcab926-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:45.445017 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.444964    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:45.445780 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.445755    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-workload-socket" pod "56528ce3-95f8-408d-96db-04ac7bcab926" (UID: "56528ce3-95f8-408d-96db-04ac7bcab926"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:45.448620 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.446498    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56528ce3-95f8-408d-96db-04ac7bcab926-istio-podinfo" pod "56528ce3-95f8-408d-96db-04ac7bcab926" (UID: "56528ce3-95f8-408d-96db-04ac7bcab926"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:01:45.448800 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.446717    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-credential-socket" pod "56528ce3-95f8-408d-96db-04ac7bcab926" (UID: "56528ce3-95f8-408d-96db-04ac7bcab926"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:45.448800 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.446729    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-istio-data" pod "56528ce3-95f8-408d-96db-04ac7bcab926" (UID: "56528ce3-95f8-408d-96db-04ac7bcab926"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:45.448800 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.448525    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56528ce3-95f8-408d-96db-04ac7bcab926-istio-token" pod "56528ce3-95f8-408d-96db-04ac7bcab926" (UID: "56528ce3-95f8-408d-96db-04ac7bcab926"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:45.450269 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.449238    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56528ce3-95f8-408d-96db-04ac7bcab926-kube-api-access-vnhhn" pod "56528ce3-95f8-408d-96db-04ac7bcab926" (UID: "56528ce3-95f8-408d-96db-04ac7bcab926"). InnerVolumeSpecName "kube-api-access-vnhhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:45.451383 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.451358    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-istio-envoy" pod "56528ce3-95f8-408d-96db-04ac7bcab926" (UID: "56528ce3-95f8-408d-96db-04ac7bcab926"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:01:45.546855 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.546824    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:45.546855 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.546851    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:45.546855 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.546866    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:45.547111 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.546875    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56528ce3-95f8-408d-96db-04ac7bcab926-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:45.547111 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.546884    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56528ce3-95f8-408d-96db-04ac7bcab926-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:45.547111 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.546893    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnhhn\" (UniqueName: \"kubernetes.io/projected/56528ce3-95f8-408d-96db-04ac7bcab926-kube-api-access-vnhhn\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:45.547111 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:45.546902    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56528ce3-95f8-408d-96db-04ac7bcab926-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:46.069717 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:46.069687    5591 generic.go:356] "Generic (PLEG): container finished" podID="56528ce3-95f8-408d-96db-04ac7bcab926" containerID="3b4385c98ae8bb6442e29c7c255251d46546c8adb221f4cacd1450be56f406f5" exitCode=0
Nov 03 20:01:46.069899 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:46.069745    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"
Nov 03 20:01:46.069899 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:46.069776    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x" event={"ID":"56528ce3-95f8-408d-96db-04ac7bcab926","Type":"ContainerDied","Data":"3b4385c98ae8bb6442e29c7c255251d46546c8adb221f4cacd1450be56f406f5"}
Nov 03 20:01:46.069899 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:46.069832    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x" event={"ID":"56528ce3-95f8-408d-96db-04ac7bcab926","Type":"ContainerDied","Data":"dd1baebcc030cf98d87d15fdb8a78557c99cec4e26dc6094923669907bfbe9e9"}
Nov 03 20:01:46.069899 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:46.069853    5591 scope.go:122] "RemoveContainer" containerID="3b4385c98ae8bb6442e29c7c255251d46546c8adb221f4cacd1450be56f406f5"
Nov 03 20:01:46.081694 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:46.081678    5591 scope.go:122] "RemoveContainer" containerID="bc51a650762e67653343b8311b56e1151c2fbe39cb177e5f812fc9753ed3be0e"
Nov 03 20:01:46.091813 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:46.091792    5591 scope.go:122] "RemoveContainer" containerID="3b4385c98ae8bb6442e29c7c255251d46546c8adb221f4cacd1450be56f406f5"
Nov 03 20:01:46.092111 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:01:46.092091    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b4385c98ae8bb6442e29c7c255251d46546c8adb221f4cacd1450be56f406f5\": container with ID starting with 3b4385c98ae8bb6442e29c7c255251d46546c8adb221f4cacd1450be56f406f5 not found: ID does not exist" containerID="3b4385c98ae8bb6442e29c7c255251d46546c8adb221f4cacd1450be56f406f5"
Nov 03 20:01:46.092164 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:46.092121    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b4385c98ae8bb6442e29c7c255251d46546c8adb221f4cacd1450be56f406f5"} err="failed to get container status \"3b4385c98ae8bb6442e29c7c255251d46546c8adb221f4cacd1450be56f406f5\": rpc error: code = NotFound desc = could not find container \"3b4385c98ae8bb6442e29c7c255251d46546c8adb221f4cacd1450be56f406f5\": container with ID starting with 3b4385c98ae8bb6442e29c7c255251d46546c8adb221f4cacd1450be56f406f5 not found: ID does not exist"
Nov 03 20:01:46.092164 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:46.092139    5591 scope.go:122] "RemoveContainer" containerID="bc51a650762e67653343b8311b56e1151c2fbe39cb177e5f812fc9753ed3be0e"
Nov 03 20:01:46.092472 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:01:46.092449    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc51a650762e67653343b8311b56e1151c2fbe39cb177e5f812fc9753ed3be0e\": container with ID starting with bc51a650762e67653343b8311b56e1151c2fbe39cb177e5f812fc9753ed3be0e not found: ID does not exist" containerID="bc51a650762e67653343b8311b56e1151c2fbe39cb177e5f812fc9753ed3be0e"
Nov 03 20:01:46.092560 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:46.092477    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc51a650762e67653343b8311b56e1151c2fbe39cb177e5f812fc9753ed3be0e"} err="failed to get container status \"bc51a650762e67653343b8311b56e1151c2fbe39cb177e5f812fc9753ed3be0e\": rpc error: code = NotFound desc = could not find container \"bc51a650762e67653343b8311b56e1151c2fbe39cb177e5f812fc9753ed3be0e\": container with ID starting with bc51a650762e67653343b8311b56e1151c2fbe39cb177e5f812fc9753ed3be0e not found: ID does not exist"
Nov 03 20:01:46.097156 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:46.097135    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"]
Nov 03 20:01:46.099963 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:46.099939    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8b3020-openshift-default-597ff96f5-qfw9x"]
Nov 03 20:01:46.471854 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:46.471810    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2"
Nov 03 20:01:46.565385 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:46.565344    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/836a78aa-c8d3-47c2-8b42-3bfd919fa477-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/836a78aa-c8d3-47c2-8b42-3bfd919fa477-plugins-config-volume\") pod \"836a78aa-c8d3-47c2-8b42-3bfd919fa477\" (UID: \"836a78aa-c8d3-47c2-8b42-3bfd919fa477\") "
Nov 03 20:01:46.565583 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:46.565483    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/836a78aa-c8d3-47c2-8b42-3bfd919fa477-kube-api-access-vggnt\" (UniqueName: \"kubernetes.io/projected/836a78aa-c8d3-47c2-8b42-3bfd919fa477-kube-api-access-vggnt\") pod \"836a78aa-c8d3-47c2-8b42-3bfd919fa477\" (UID: \"836a78aa-c8d3-47c2-8b42-3bfd919fa477\") "
Nov 03 20:01:46.565814 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:46.565793    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/836a78aa-c8d3-47c2-8b42-3bfd919fa477-plugins-config-volume" pod "836a78aa-c8d3-47c2-8b42-3bfd919fa477" (UID: "836a78aa-c8d3-47c2-8b42-3bfd919fa477"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:01:46.566229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:46.566193    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/836a78aa-c8d3-47c2-8b42-3bfd919fa477-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:46.567967 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:46.567938    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/836a78aa-c8d3-47c2-8b42-3bfd919fa477-kube-api-access-vggnt" pod "836a78aa-c8d3-47c2-8b42-3bfd919fa477" (UID: "836a78aa-c8d3-47c2-8b42-3bfd919fa477"). InnerVolumeSpecName "kube-api-access-vggnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:46.668055 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:46.667958    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vggnt\" (UniqueName: \"kubernetes.io/projected/836a78aa-c8d3-47c2-8b42-3bfd919fa477-kube-api-access-vggnt\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:47.079779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.079746    5591 generic.go:356] "Generic (PLEG): container finished" podID="836a78aa-c8d3-47c2-8b42-3bfd919fa477" containerID="5ec71ebb619cbe952e9d9ca2c58e395ad74d10b9a157b2a1480f77f7167cad86" exitCode=0
Nov 03 20:01:47.079994 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.079792    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2" event={"ID":"836a78aa-c8d3-47c2-8b42-3bfd919fa477","Type":"ContainerDied","Data":"5ec71ebb619cbe952e9d9ca2c58e395ad74d10b9a157b2a1480f77f7167cad86"}
Nov 03 20:01:47.079994 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.079810    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2"
Nov 03 20:01:47.079994 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.079873    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2" event={"ID":"836a78aa-c8d3-47c2-8b42-3bfd919fa477","Type":"ContainerDied","Data":"5290978dceb5a2a42635aecb056c5b639e0a2f29f8fc7b395054f216caf1d9cb"}
Nov 03 20:01:47.079994 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.079901    5591 scope.go:122] "RemoveContainer" containerID="5ec71ebb619cbe952e9d9ca2c58e395ad74d10b9a157b2a1480f77f7167cad86"
Nov 03 20:01:47.090701 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.090679    5591 scope.go:122] "RemoveContainer" containerID="5ec71ebb619cbe952e9d9ca2c58e395ad74d10b9a157b2a1480f77f7167cad86"
Nov 03 20:01:47.090964 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:01:47.090942    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ec71ebb619cbe952e9d9ca2c58e395ad74d10b9a157b2a1480f77f7167cad86\": container with ID starting with 5ec71ebb619cbe952e9d9ca2c58e395ad74d10b9a157b2a1480f77f7167cad86 not found: ID does not exist" containerID="5ec71ebb619cbe952e9d9ca2c58e395ad74d10b9a157b2a1480f77f7167cad86"
Nov 03 20:01:47.091024 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.090975    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ec71ebb619cbe952e9d9ca2c58e395ad74d10b9a157b2a1480f77f7167cad86"} err="failed to get container status \"5ec71ebb619cbe952e9d9ca2c58e395ad74d10b9a157b2a1480f77f7167cad86\": rpc error: code = NotFound desc = could not find container \"5ec71ebb619cbe952e9d9ca2c58e395ad74d10b9a157b2a1480f77f7167cad86\": container with ID starting with 5ec71ebb619cbe952e9d9ca2c58e395ad74d10b9a157b2a1480f77f7167cad86 not found: ID does not exist"
Nov 03 20:01:47.104355 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.104335    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2"]
Nov 03 20:01:47.108260 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.108239    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-1b1d04-5d49fb46ff-xtnb2"]
Nov 03 20:01:47.350454 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.350362    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56528ce3-95f8-408d-96db-04ac7bcab926" path="/var/lib/kubelet/pods/56528ce3-95f8-408d-96db-04ac7bcab926/volumes"
Nov 03 20:01:47.350935 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.350913    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="836a78aa-c8d3-47c2-8b42-3bfd919fa477" path="/var/lib/kubelet/pods/836a78aa-c8d3-47c2-8b42-3bfd919fa477/volumes"
Nov 03 20:01:47.931227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.926927    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"]
Nov 03 20:01:47.931227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.927922    5591 cpu_manager.go:414] "containerMap: removing container" podUID="836a78aa-c8d3-47c2-8b42-3bfd919fa477" containerName="payload-pre-processing"
Nov 03 20:01:47.931227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.927940    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="836a78aa-c8d3-47c2-8b42-3bfd919fa477" containerName="payload-pre-processing"
Nov 03 20:01:47.931227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.927965    5591 cpu_manager.go:414] "containerMap: removing container" podUID="56528ce3-95f8-408d-96db-04ac7bcab926" containerName="istio-proxy"
Nov 03 20:01:47.931227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.927973    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56528ce3-95f8-408d-96db-04ac7bcab926" containerName="istio-proxy"
Nov 03 20:01:47.931227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.928017    5591 cpu_manager.go:414] "containerMap: removing container" podUID="56528ce3-95f8-408d-96db-04ac7bcab926" containerName="istio-proxy"
Nov 03 20:01:47.931227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.928024    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56528ce3-95f8-408d-96db-04ac7bcab926" containerName="istio-proxy"
Nov 03 20:01:47.931227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.928183    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="836a78aa-c8d3-47c2-8b42-3bfd919fa477" containerName="payload-pre-processing"
Nov 03 20:01:47.933975 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.933943    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="56528ce3-95f8-408d-96db-04ac7bcab926" containerName="istio-proxy"
Nov 03 20:01:47.934126 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.934004    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="56528ce3-95f8-408d-96db-04ac7bcab926" containerName="istio-proxy"
Nov 03 20:01:47.940358 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.940330    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:47.942398 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.942374    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"]
Nov 03 20:01:47.990879 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.990839    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:47.990879 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.990877    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:47.991164 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.990958    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr6s4\" (UniqueName: \"kubernetes.io/projected/ac8f5ae3-f947-438b-a93b-05bcdb362154-kube-api-access-qr6s4\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:47.991164 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.991031    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac8f5ae3-f947-438b-a93b-05bcdb362154-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:47.991164 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.991058    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:47.991164 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.991084    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:47.991164 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.991099    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:47.991164 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.991165    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:47.991582 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:47.991187    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:48.092851 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:48.092819    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac8f5ae3-f947-438b-a93b-05bcdb362154-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:48.093026 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:48.092859    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:48.093026 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:48.092984    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:48.093026 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:48.093022    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:48.093150 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:48.093132    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:48.093231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:48.093155    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:48.093280 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:48.093250    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:48.093335 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:48.093304    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:48.093335 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:48.093326    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:48.093435 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:48.093414    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:48.093491 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:48.093450    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr6s4\" (UniqueName: \"kubernetes.io/projected/ac8f5ae3-f947-438b-a93b-05bcdb362154-kube-api-access-qr6s4\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:48.093635 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:48.093611    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac8f5ae3-f947-438b-a93b-05bcdb362154-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:48.093897 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:48.093676    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:48.093980 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:48.093912    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:48.095803 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:48.095782    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:48.095863 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:48.095791    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:48.105801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:48.105772    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:48.105981 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:48.105961    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr6s4\" (UniqueName: \"kubernetes.io/projected/ac8f5ae3-f947-438b-a93b-05bcdb362154-kube-api-access-qr6s4\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:48.262658 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:48.262575    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:48.563975 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:48.563946    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"]
Nov 03 20:01:48.564840 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:01:48.564813    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac8f5ae3_f947_438b_a93b_05bcdb362154.slice/crio-454765e2179b815c6068fee12087a79a07eda47f05a376c095d6da04852b1638 WatchSource:0}: Error finding container 454765e2179b815c6068fee12087a79a07eda47f05a376c095d6da04852b1638: Status 404 returned error can't find the container with id 454765e2179b815c6068fee12087a79a07eda47f05a376c095d6da04852b1638
Nov 03 20:01:48.567822 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:48.567786    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:01:48.567903 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:48.567857    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:01:48.567903 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:48.567893    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:01:49.143229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:49.139303    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47" event={"ID":"ac8f5ae3-f947-438b-a93b-05bcdb362154","Type":"ContainerStarted","Data":"cbfec5d0839f38a2420161072e3f5588585ced998bf563158007ce7b4d35d833"}
Nov 03 20:01:49.143229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:49.139390    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47" event={"ID":"ac8f5ae3-f947-438b-a93b-05bcdb362154","Type":"ContainerStarted","Data":"454765e2179b815c6068fee12087a79a07eda47f05a376c095d6da04852b1638"}
Nov 03 20:01:49.265508 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:49.265477    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:01:49.265816 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:49.265795    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:01:49.265999 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:49.265971    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47" podUID="ac8f5ae3-f947-438b-a93b-05bcdb362154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 03 20:01:50.001257 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:49.996313    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:50.050230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:50.049304    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 20:01:50.146192 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:50.142905    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:50.173856 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:50.173808    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:50.198236 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:50.190031    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47" podStartSLOduration=3.190009735 podStartE2EDuration="3.190009735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:49.181161413 +0000 UTC m=+1457.953454432" watchObservedRunningTime="2026-11-03 20:01:50.190009735 +0000 UTC m=+1458.962302753"
Nov 03 20:01:50.205356 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:50.205289    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq"]
Nov 03 20:01:50.218296 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:50.215689    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c"]
Nov 03 20:01:50.409664 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:50.408159    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 20:01:50.409664 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:50.409110    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:01:50.409664 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:50.409155    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47" podUID="ac8f5ae3-f947-438b-a93b-05bcdb362154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 03 20:01:50.655270 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:50.654768    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:50.825397 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:50.753638    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:51.028432 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:51.028393    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:51.081923 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:51.081807    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:51.089986 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:51.089948    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 20:01:51.167466 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:51.167423    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 20:01:51.589345 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:51.173334    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:51.822041 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:51.405405    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:01:51.893341 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:51.428694    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:01:51.893978 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:51.893849    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47" podUID="ac8f5ae3-f947-438b-a93b-05bcdb362154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 03 20:01:51.894288 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:51.666269    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:01:51.894445 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:51.781897    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 20:01:51.894560 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:51.829414    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:51.894676 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:51.443330    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 20:01:52.039232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:52.035305    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:52.111454 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:52.111416    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 20:01:52.142402 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:52.142362    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 20:01:52.142803 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:52.142784    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 20:01:52.143012 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:52.142989    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 20:01:52.143171 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:52.143158    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 20:01:52.267887 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:52.267842    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 20:01:52.408758 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:52.408719    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:01:52.409047 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:52.409017    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47" podUID="ac8f5ae3-f947-438b-a93b-05bcdb362154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 03 20:01:53.008227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:52.982221    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 20:01:53.098021 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:53.096348    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:01:53.098455 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:53.096393    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:53.122937 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:53.096437    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:53.124581 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:53.096469    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:53.125076 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:53.096487    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 20:01:53.285291 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:53.285258    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:01:53.285912 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:53.285865    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47" podUID="ac8f5ae3-f947-438b-a93b-05bcdb362154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 03 20:01:53.332291 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:53.304754    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:01:53.524428 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:53.524398    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 20:01:53.524725 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:53.524709    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 20:01:53.524888 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:53.524876    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 20:01:53.525019 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:53.524998    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 20:01:53.548136 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:53.540437    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 20:01:53.671226 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:53.670376    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 20:01:54.263908 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:54.263867    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:01:54.264091 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:54.263943    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47" podUID="ac8f5ae3-f947-438b-a93b-05bcdb362154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 03 20:01:54.770023 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:54.769981    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq"]
Nov 03 20:01:54.770444 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:54.770266    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq" podUID="fdab0164-a331-4abe-a4ed-f5c299bde686" containerName="maas-api" containerID="cri-o://45d155c9f013036b320c35c58b5f1b4724771e46fa5b5f79cc708c18747b90d9" gracePeriod=30
Nov 03 20:01:55.243259 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.242396    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c" podUID="7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2" containerName="payload-pre-processing" containerID="cri-o://2a889f04506b841873ec3c39c56a9370004baa0db49b8407088e49f46a878c2c" gracePeriod=25
Nov 03 20:01:55.245172 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.245140    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq" podUID="d9c2a580-9312-4873-8b6d-f8970cab6e18" containerName="payload-processing" containerID="cri-o://94eac53a7af7d417f6b5c3a47cce639c12b849302653293dc6c516ce2e213365" gracePeriod=25
Nov 03 20:01:55.268582 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.263636    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:01:55.268582 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.263687    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47" podUID="ac8f5ae3-f947-438b-a93b-05bcdb362154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 03 20:01:55.452387 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.447843    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq"
Nov 03 20:01:55.683278 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.659074    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdab0164-a331-4abe-a4ed-f5c299bde686-kube-api-access-pzgg8\" (UniqueName: \"kubernetes.io/projected/fdab0164-a331-4abe-a4ed-f5c299bde686-kube-api-access-pzgg8\") pod \"fdab0164-a331-4abe-a4ed-f5c299bde686\" (UID: \"fdab0164-a331-4abe-a4ed-f5c299bde686\") "
Nov 03 20:01:55.757304 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.714986    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdab0164-a331-4abe-a4ed-f5c299bde686-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdab0164-a331-4abe-a4ed-f5c299bde686-maas-api-tls\") pod \"fdab0164-a331-4abe-a4ed-f5c299bde686\" (UID: \"fdab0164-a331-4abe-a4ed-f5c299bde686\") "
Nov 03 20:01:55.757304 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.745818    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdab0164-a331-4abe-a4ed-f5c299bde686-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdab0164-a331-4abe-a4ed-f5c299bde686-openshift-service-ca\") pod \"fdab0164-a331-4abe-a4ed-f5c299bde686\" (UID: \"fdab0164-a331-4abe-a4ed-f5c299bde686\") "
Nov 03 20:01:55.757304 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.745896    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdab0164-a331-4abe-a4ed-f5c299bde686-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdab0164-a331-4abe-a4ed-f5c299bde686-metrics-certs\") pod \"fdab0164-a331-4abe-a4ed-f5c299bde686\" (UID: \"fdab0164-a331-4abe-a4ed-f5c299bde686\") "
Nov 03 20:01:55.757304 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.716410    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdab0164-a331-4abe-a4ed-f5c299bde686-kube-api-access-pzgg8" pod "fdab0164-a331-4abe-a4ed-f5c299bde686" (UID: "fdab0164-a331-4abe-a4ed-f5c299bde686"). InnerVolumeSpecName "kube-api-access-pzgg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:55.757304 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.751852    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdab0164-a331-4abe-a4ed-f5c299bde686-openshift-service-ca" pod "fdab0164-a331-4abe-a4ed-f5c299bde686" (UID: "fdab0164-a331-4abe-a4ed-f5c299bde686"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:01:55.757304 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.751930    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdab0164-a331-4abe-a4ed-f5c299bde686-metrics-certs" pod "fdab0164-a331-4abe-a4ed-f5c299bde686" (UID: "fdab0164-a331-4abe-a4ed-f5c299bde686"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:01:55.761523 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.761471    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdab0164-a331-4abe-a4ed-f5c299bde686-maas-api-tls" pod "fdab0164-a331-4abe-a4ed-f5c299bde686" (UID: "fdab0164-a331-4abe-a4ed-f5c299bde686"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:01:55.845460 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.845045    5591 generic.go:356] "Generic (PLEG): container finished" podID="fdab0164-a331-4abe-a4ed-f5c299bde686" containerID="45d155c9f013036b320c35c58b5f1b4724771e46fa5b5f79cc708c18747b90d9" exitCode=0
Nov 03 20:01:55.845460 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.845177    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq" event={"ID":"fdab0164-a331-4abe-a4ed-f5c299bde686","Type":"ContainerDied","Data":"45d155c9f013036b320c35c58b5f1b4724771e46fa5b5f79cc708c18747b90d9"}
Nov 03 20:01:55.845460 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.845263    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq" event={"ID":"fdab0164-a331-4abe-a4ed-f5c299bde686","Type":"ContainerDied","Data":"7362b3afca0aec3643c24e4264cba9e30a129b1270d797f0101ac55ea0aa3fa0"}
Nov 03 20:01:55.845460 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.845287    5591 scope.go:122] "RemoveContainer" containerID="45d155c9f013036b320c35c58b5f1b4724771e46fa5b5f79cc708c18747b90d9"
Nov 03 20:01:55.846153 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.846088    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq"
Nov 03 20:01:55.850569 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.850545    5591 generic.go:356] "Generic (PLEG): container finished" podID="7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2" containerID="2a889f04506b841873ec3c39c56a9370004baa0db49b8407088e49f46a878c2c" exitCode=0
Nov 03 20:01:55.850826 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.850805    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c" event={"ID":"7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2","Type":"ContainerDied","Data":"2a889f04506b841873ec3c39c56a9370004baa0db49b8407088e49f46a878c2c"}
Nov 03 20:01:55.853298 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.853266    5591 generic.go:356] "Generic (PLEG): container finished" podID="d9c2a580-9312-4873-8b6d-f8970cab6e18" containerID="94eac53a7af7d417f6b5c3a47cce639c12b849302653293dc6c516ce2e213365" exitCode=0
Nov 03 20:01:55.853408 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.853394    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq" event={"ID":"d9c2a580-9312-4873-8b6d-f8970cab6e18","Type":"ContainerDied","Data":"94eac53a7af7d417f6b5c3a47cce639c12b849302653293dc6c516ce2e213365"}
Nov 03 20:01:55.854322 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.854301    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzgg8\" (UniqueName: \"kubernetes.io/projected/fdab0164-a331-4abe-a4ed-f5c299bde686-kube-api-access-pzgg8\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:55.854322 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.854324    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdab0164-a331-4abe-a4ed-f5c299bde686-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:55.854502 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.854335    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdab0164-a331-4abe-a4ed-f5c299bde686-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:55.854502 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.854343    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdab0164-a331-4abe-a4ed-f5c299bde686-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:55.872163 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.872137    5591 scope.go:122] "RemoveContainer" containerID="45d155c9f013036b320c35c58b5f1b4724771e46fa5b5f79cc708c18747b90d9"
Nov 03 20:01:55.872473 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.872451    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c"
Nov 03 20:01:55.872580 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:01:55.872493    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45d155c9f013036b320c35c58b5f1b4724771e46fa5b5f79cc708c18747b90d9\": container with ID starting with 45d155c9f013036b320c35c58b5f1b4724771e46fa5b5f79cc708c18747b90d9 not found: ID does not exist" containerID="45d155c9f013036b320c35c58b5f1b4724771e46fa5b5f79cc708c18747b90d9"
Nov 03 20:01:55.872580 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.872525    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d155c9f013036b320c35c58b5f1b4724771e46fa5b5f79cc708c18747b90d9"} err="failed to get container status \"45d155c9f013036b320c35c58b5f1b4724771e46fa5b5f79cc708c18747b90d9\": rpc error: code = NotFound desc = could not find container \"45d155c9f013036b320c35c58b5f1b4724771e46fa5b5f79cc708c18747b90d9\": container with ID starting with 45d155c9f013036b320c35c58b5f1b4724771e46fa5b5f79cc708c18747b90d9 not found: ID does not exist"
Nov 03 20:01:55.929258 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.918436    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq"]
Nov 03 20:01:55.929258 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.921787    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f493f7-5f565cf4bf-gr5qq"]
Nov 03 20:01:55.957048 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.956964    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2-plugins-config-volume\") pod \"7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2\" (UID: \"7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2\") "
Nov 03 20:01:55.957244 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.957227    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2-kube-api-access-9ntn2\" (UniqueName: \"kubernetes.io/projected/7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2-kube-api-access-9ntn2\") pod \"7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2\" (UID: \"7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2\") "
Nov 03 20:01:55.958382 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.958354    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2-plugins-config-volume" pod "7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2" (UID: "7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:01:55.962333 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:55.962304    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2-kube-api-access-9ntn2" pod "7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2" (UID: "7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2"). InnerVolumeSpecName "kube-api-access-9ntn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:56.024185 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.024158    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq"
Nov 03 20:01:56.058446 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.058417    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9c2a580-9312-4873-8b6d-f8970cab6e18-kube-api-access-lqw7v\" (UniqueName: \"kubernetes.io/projected/d9c2a580-9312-4873-8b6d-f8970cab6e18-kube-api-access-lqw7v\") pod \"d9c2a580-9312-4873-8b6d-f8970cab6e18\" (UID: \"d9c2a580-9312-4873-8b6d-f8970cab6e18\") "
Nov 03 20:01:56.058683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.058662    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9c2a580-9312-4873-8b6d-f8970cab6e18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c2a580-9312-4873-8b6d-f8970cab6e18-plugins-config-volume\") pod \"d9c2a580-9312-4873-8b6d-f8970cab6e18\" (UID: \"d9c2a580-9312-4873-8b6d-f8970cab6e18\") "
Nov 03 20:01:56.059113 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.059082    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c2a580-9312-4873-8b6d-f8970cab6e18-plugins-config-volume" pod "d9c2a580-9312-4873-8b6d-f8970cab6e18" (UID: "d9c2a580-9312-4873-8b6d-f8970cab6e18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:01:56.059374 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.059353    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ntn2\" (UniqueName: \"kubernetes.io/projected/7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2-kube-api-access-9ntn2\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:56.059374 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.059379    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c2a580-9312-4873-8b6d-f8970cab6e18-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:56.059374 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.059395    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:56.060687 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.060664    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c2a580-9312-4873-8b6d-f8970cab6e18-kube-api-access-lqw7v" pod "d9c2a580-9312-4873-8b6d-f8970cab6e18" (UID: "d9c2a580-9312-4873-8b6d-f8970cab6e18"). InnerVolumeSpecName "kube-api-access-lqw7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:01:56.160414 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.160369    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqw7v\" (UniqueName: \"kubernetes.io/projected/d9c2a580-9312-4873-8b6d-f8970cab6e18-kube-api-access-lqw7v\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:01:56.263096 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.263023    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:01:56.263096 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.263085    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47" podUID="ac8f5ae3-f947-438b-a93b-05bcdb362154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 03 20:01:56.639100 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.639063    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd"]
Nov 03 20:01:56.639735 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.639719    5591 cpu_manager.go:414] "containerMap: removing container" podUID="d9c2a580-9312-4873-8b6d-f8970cab6e18" containerName="payload-processing"
Nov 03 20:01:56.639735 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.639735    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9c2a580-9312-4873-8b6d-f8970cab6e18" containerName="payload-processing"
Nov 03 20:01:56.639836 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.639760    5591 cpu_manager.go:414] "containerMap: removing container" podUID="fdab0164-a331-4abe-a4ed-f5c299bde686" containerName="maas-api"
Nov 03 20:01:56.639836 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.639766    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdab0164-a331-4abe-a4ed-f5c299bde686" containerName="maas-api"
Nov 03 20:01:56.639836 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.639776    5591 cpu_manager.go:414] "containerMap: removing container" podUID="7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2" containerName="payload-pre-processing"
Nov 03 20:01:56.639836 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.639781    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2" containerName="payload-pre-processing"
Nov 03 20:01:56.639960 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.639886    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdab0164-a331-4abe-a4ed-f5c299bde686" containerName="maas-api"
Nov 03 20:01:56.639960 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.639896    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2" containerName="payload-pre-processing"
Nov 03 20:01:56.639960 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.639909    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9c2a580-9312-4873-8b6d-f8970cab6e18" containerName="payload-processing"
Nov 03 20:01:56.643416 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.643400    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd"
Nov 03 20:01:56.653183 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.653146    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-8fa615\""
Nov 03 20:01:56.654986 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.654963    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd"]
Nov 03 20:01:56.673996 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.673967    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts"]
Nov 03 20:01:56.707617 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.707580    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts"
Nov 03 20:01:56.713066 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.713031    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts"]
Nov 03 20:01:56.720570 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.720548    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-8fa615-dockercfg-jhjk4\""
Nov 03 20:01:56.816458 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.816396    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbkmc\" (UniqueName: \"kubernetes.io/projected/1837d3e9-0dc6-4e35-b915-688a59b244bc-kube-api-access-fbkmc\") pod \"payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd\" (UID: \"1837d3e9-0dc6-4e35-b915-688a59b244bc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd"
Nov 03 20:01:56.820232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.816643    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7b5608-93ed-41f2-a0ed-995d6041bf70-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts\" (UID: \"bb7b5608-93ed-41f2-a0ed-995d6041bf70\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts"
Nov 03 20:01:56.820232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.816686    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzxbh\" (UniqueName: \"kubernetes.io/projected/bb7b5608-93ed-41f2-a0ed-995d6041bf70-kube-api-access-zzxbh\") pod \"payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts\" (UID: \"bb7b5608-93ed-41f2-a0ed-995d6041bf70\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts"
Nov 03 20:01:56.820232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.816745    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1837d3e9-0dc6-4e35-b915-688a59b244bc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd\" (UID: \"1837d3e9-0dc6-4e35-b915-688a59b244bc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd"
Nov 03 20:01:56.952294 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.949784    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c" event={"ID":"7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2","Type":"ContainerDied","Data":"02c4a500eef3187c74d89f0bfcb1b5c16c4644767bef991fd05f4209e2f12586"}
Nov 03 20:01:56.952294 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.949897    5591 scope.go:122] "RemoveContainer" containerID="2a889f04506b841873ec3c39c56a9370004baa0db49b8407088e49f46a878c2c"
Nov 03 20:01:56.952294 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.950100    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c"
Nov 03 20:01:56.952997 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.952648    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbkmc\" (UniqueName: \"kubernetes.io/projected/1837d3e9-0dc6-4e35-b915-688a59b244bc-kube-api-access-fbkmc\") pod \"payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd\" (UID: \"1837d3e9-0dc6-4e35-b915-688a59b244bc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd"
Nov 03 20:01:56.952997 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.952961    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7b5608-93ed-41f2-a0ed-995d6041bf70-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts\" (UID: \"bb7b5608-93ed-41f2-a0ed-995d6041bf70\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts"
Nov 03 20:01:56.953114 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.952998    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzxbh\" (UniqueName: \"kubernetes.io/projected/bb7b5608-93ed-41f2-a0ed-995d6041bf70-kube-api-access-zzxbh\") pod \"payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts\" (UID: \"bb7b5608-93ed-41f2-a0ed-995d6041bf70\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts"
Nov 03 20:01:56.953114 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.953065    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1837d3e9-0dc6-4e35-b915-688a59b244bc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd\" (UID: \"1837d3e9-0dc6-4e35-b915-688a59b244bc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd"
Nov 03 20:01:56.960234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.957742    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1837d3e9-0dc6-4e35-b915-688a59b244bc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd\" (UID: \"1837d3e9-0dc6-4e35-b915-688a59b244bc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd"
Nov 03 20:01:56.960234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.958519    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7b5608-93ed-41f2-a0ed-995d6041bf70-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts\" (UID: \"bb7b5608-93ed-41f2-a0ed-995d6041bf70\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts"
Nov 03 20:01:56.974742 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.961416    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq" event={"ID":"d9c2a580-9312-4873-8b6d-f8970cab6e18","Type":"ContainerDied","Data":"ceb00877cb11eccdb33607b39d3e89b424bd08eb61c0b00a5dac43ed9ebf8f9e"}
Nov 03 20:01:56.974742 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:56.961523    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq"
Nov 03 20:01:57.113495 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:57.112565    5591 scope.go:122] "RemoveContainer" containerID="94eac53a7af7d417f6b5c3a47cce639c12b849302653293dc6c516ce2e213365"
Nov 03 20:01:57.113658 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:57.112883    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbkmc\" (UniqueName: \"kubernetes.io/projected/1837d3e9-0dc6-4e35-b915-688a59b244bc-kube-api-access-fbkmc\") pod \"payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd\" (UID: \"1837d3e9-0dc6-4e35-b915-688a59b244bc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd"
Nov 03 20:01:57.115060 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:57.114988    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzxbh\" (UniqueName: \"kubernetes.io/projected/bb7b5608-93ed-41f2-a0ed-995d6041bf70-kube-api-access-zzxbh\") pod \"payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts\" (UID: \"bb7b5608-93ed-41f2-a0ed-995d6041bf70\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts"
Nov 03 20:01:57.159803 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:57.159779    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq"]
Nov 03 20:01:57.164595 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:57.164572    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f493f7-79fd656c6-s28fq"]
Nov 03 20:01:57.205292 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:57.204852    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c"]
Nov 03 20:01:57.208355 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:57.208328    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f493f7-8bb6b74c-nk64c"]
Nov 03 20:01:57.259612 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:57.259580    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd"
Nov 03 20:01:57.262993 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:57.262964    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:01:57.263111 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:57.263028    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47" podUID="ac8f5ae3-f947-438b-a93b-05bcdb362154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 03 20:01:57.325863 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:57.325826    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts"
Nov 03 20:01:57.469332 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:57.381508    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2" path="/var/lib/kubelet/pods/7e13f9b7-9144-4bc4-9ce8-5a9d3d3641b2/volumes"
Nov 03 20:01:57.469332 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:57.382167    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9c2a580-9312-4873-8b6d-f8970cab6e18" path="/var/lib/kubelet/pods/d9c2a580-9312-4873-8b6d-f8970cab6e18/volumes"
Nov 03 20:01:57.469332 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:57.468242    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdab0164-a331-4abe-a4ed-f5c299bde686" path="/var/lib/kubelet/pods/fdab0164-a331-4abe-a4ed-f5c299bde686/volumes"
Nov 03 20:01:57.842532 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:57.842506    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts"]
Nov 03 20:01:57.844333 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:01:57.844301    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7b5608_93ed_41f2_a0ed_995d6041bf70.slice/crio-4ec7e5984fb254b2baec03150a7e549f4d8baa69fcb9a5ece787e536a5f2ad9e WatchSource:0}: Error finding container 4ec7e5984fb254b2baec03150a7e549f4d8baa69fcb9a5ece787e536a5f2ad9e: Status 404 returned error can't find the container with id 4ec7e5984fb254b2baec03150a7e549f4d8baa69fcb9a5ece787e536a5f2ad9e
Nov 03 20:01:57.992813 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:57.971830    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd"]
Nov 03 20:01:58.018712 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:58.018682    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd" event={"ID":"1837d3e9-0dc6-4e35-b915-688a59b244bc","Type":"ContainerStarted","Data":"29f1652ea596e833514e3ecc2ce42681f20c0266add3a7772f30dd60abfdf244"}
Nov 03 20:01:58.024037 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:58.024006    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts" event={"ID":"bb7b5608-93ed-41f2-a0ed-995d6041bf70","Type":"ContainerStarted","Data":"4ec7e5984fb254b2baec03150a7e549f4d8baa69fcb9a5ece787e536a5f2ad9e"}
Nov 03 20:01:58.263478 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:58.263445    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:01:58.263669 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:58.263498    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47" podUID="ac8f5ae3-f947-438b-a93b-05bcdb362154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 03 20:01:58.727038 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:01:58.726999    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:01:59.048757 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:59.048719    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts" event={"ID":"bb7b5608-93ed-41f2-a0ed-995d6041bf70","Type":"ContainerStarted","Data":"70cad7d7a9903c5b58e8e2dad32cc24d8ab36d3ddfe233f24f85760124b00385"}
Nov 03 20:01:59.052558 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:59.050566    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts"
Nov 03 20:01:59.055878 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:59.055851    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd" event={"ID":"1837d3e9-0dc6-4e35-b915-688a59b244bc","Type":"ContainerStarted","Data":"9d4ad3129312dabba2f5b46d5237436340299c242b8d02c59b2d6aa850f879c2"}
Nov 03 20:01:59.057329 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:59.056923    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd"
Nov 03 20:01:59.084063 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:59.082430    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts" podStartSLOduration=3.08240981 podStartE2EDuration="3.08240981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:59.078674243 +0000 UTC m=+1467.850967261" watchObservedRunningTime="2026-11-03 20:01:59.08240981 +0000 UTC m=+1467.854702830"
Nov 03 20:01:59.121228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:59.103945    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd" podStartSLOduration=3.103921241 podStartE2EDuration="3.103921241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:01:59.10062664 +0000 UTC m=+1467.872919658" watchObservedRunningTime="2026-11-03 20:01:59.103921241 +0000 UTC m=+1467.876214291"
Nov 03 20:01:59.266795 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:59.266759    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 03 20:01:59.267102 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:01:59.267052    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47" podUID="ac8f5ae3-f947-438b-a93b-05bcdb362154" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 03 20:02:00.267364 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:00.267334    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:02:00.267861 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:00.267760    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:02:00.268730 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:00.268708    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:02:01.874894 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:01.874861    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c"]
Nov 03 20:02:01.879789 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:01.879764    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c"
Nov 03 20:02:01.889873 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:01.889844    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-8fa615-serving-cert\""
Nov 03 20:02:01.894155 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:01.894132    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c"]
Nov 03 20:02:01.976079 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:01.976045    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhh8b\" (UniqueName: \"kubernetes.io/projected/1a0b68e2-5f67-415a-96d0-8186134792bc-kube-api-access-lhh8b\") pod \"maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c\" (UID: \"1a0b68e2-5f67-415a-96d0-8186134792bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c"
Nov 03 20:02:01.976294 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:01.976266    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a0b68e2-5f67-415a-96d0-8186134792bc-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c\" (UID: \"1a0b68e2-5f67-415a-96d0-8186134792bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c"
Nov 03 20:02:01.976377 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:01.976359    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a0b68e2-5f67-415a-96d0-8186134792bc-metrics-certs\") pod \"maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c\" (UID: \"1a0b68e2-5f67-415a-96d0-8186134792bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c"
Nov 03 20:02:01.976478 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:01.976468    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a0b68e2-5f67-415a-96d0-8186134792bc-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c\" (UID: \"1a0b68e2-5f67-415a-96d0-8186134792bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c"
Nov 03 20:02:02.078317 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:02.078275    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a0b68e2-5f67-415a-96d0-8186134792bc-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c\" (UID: \"1a0b68e2-5f67-415a-96d0-8186134792bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c"
Nov 03 20:02:02.082790 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:02.082754    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhh8b\" (UniqueName: \"kubernetes.io/projected/1a0b68e2-5f67-415a-96d0-8186134792bc-kube-api-access-lhh8b\") pod \"maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c\" (UID: \"1a0b68e2-5f67-415a-96d0-8186134792bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c"
Nov 03 20:02:02.083192 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:02.083165    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a0b68e2-5f67-415a-96d0-8186134792bc-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c\" (UID: \"1a0b68e2-5f67-415a-96d0-8186134792bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c"
Nov 03 20:02:02.083402 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:02.083384    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a0b68e2-5f67-415a-96d0-8186134792bc-metrics-certs\") pod \"maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c\" (UID: \"1a0b68e2-5f67-415a-96d0-8186134792bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c"
Nov 03 20:02:02.085516 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:02.085486    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a0b68e2-5f67-415a-96d0-8186134792bc-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c\" (UID: \"1a0b68e2-5f67-415a-96d0-8186134792bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c"
Nov 03 20:02:02.104106 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:02.104071    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhh8b\" (UniqueName: \"kubernetes.io/projected/1a0b68e2-5f67-415a-96d0-8186134792bc-kube-api-access-lhh8b\") pod \"maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c\" (UID: \"1a0b68e2-5f67-415a-96d0-8186134792bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c"
Nov 03 20:02:02.118247 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:02.115589    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a0b68e2-5f67-415a-96d0-8186134792bc-maas-api-tls\") pod \"maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c\" (UID: \"1a0b68e2-5f67-415a-96d0-8186134792bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c"
Nov 03 20:02:02.130994 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:02.130351    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a0b68e2-5f67-415a-96d0-8186134792bc-metrics-certs\") pod \"maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c\" (UID: \"1a0b68e2-5f67-415a-96d0-8186134792bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c"
Nov 03 20:02:02.200248 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:02.200186    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c"
Nov 03 20:02:02.490110 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:02.490081    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c"]
Nov 03 20:02:02.491529 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:02:02.491488    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a0b68e2_5f67_415a_96d0_8186134792bc.slice/crio-9b1eb91707a435bad587305a621207edde5099120609e2fa0f66e2af3193ecf1 WatchSource:0}: Error finding container 9b1eb91707a435bad587305a621207edde5099120609e2fa0f66e2af3193ecf1: Status 404 returned error can't find the container with id 9b1eb91707a435bad587305a621207edde5099120609e2fa0f66e2af3193ecf1
Nov 03 20:02:03.100968 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:03.100933    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c" event={"ID":"1a0b68e2-5f67-415a-96d0-8186134792bc","Type":"ContainerStarted","Data":"8b987c3f3c8ee227f46723ca2c8809b1b8f59b99eeb21393b39233e165192723"}
Nov 03 20:02:03.101442 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:03.100992    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c" event={"ID":"1a0b68e2-5f67-415a-96d0-8186134792bc","Type":"ContainerStarted","Data":"9b1eb91707a435bad587305a621207edde5099120609e2fa0f66e2af3193ecf1"}
Nov 03 20:02:03.101442 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:03.101115    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c"
Nov 03 20:02:03.123402 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:03.123348    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c" podStartSLOduration=1.6038492290000002 podStartE2EDuration="2.123333856s" totalImagesPullingTime="519.484627ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:03.120256212 +0000 UTC m=+1471.892549230" watchObservedRunningTime="2026-11-03 20:02:03.123333856 +0000 UTC m=+1471.895626871"
Nov 03 20:02:07.541947 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:07.541910    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:02:09.116011 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:09.115979    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c"
Nov 03 20:02:10.673566 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:10.673531    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"]
Nov 03 20:02:10.674061 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:10.673767    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn" podUID="3a0f28bc-d05f-45bb-8431-3aaf73529510" containerName="istio-proxy" containerID="cri-o://00bc6fbf7920866477e22c63b8322d9015bf9059e8424f6465b6eb5fe6a77e8e" gracePeriod=30
Nov 03 20:02:11.072806 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:11.072780    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd"
Nov 03 20:02:11.073065 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:11.073046    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts"
Nov 03 20:02:16.029633 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.029608    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 20:02:16.145527 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.145494    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a0f28bc-d05f-45bb-8431-3aaf73529510-kube-api-access-slwhg\" (UniqueName: \"kubernetes.io/projected/3a0f28bc-d05f-45bb-8431-3aaf73529510-kube-api-access-slwhg\") pod \"3a0f28bc-d05f-45bb-8431-3aaf73529510\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") "
Nov 03 20:02:16.145527 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.145529    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-token\" (UniqueName: \"kubernetes.io/projected/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-token\") pod \"3a0f28bc-d05f-45bb-8431-3aaf73529510\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") "
Nov 03 20:02:16.145782 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.145562    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-data\") pod \"3a0f28bc-d05f-45bb-8431-3aaf73529510\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") "
Nov 03 20:02:16.145782 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.145599    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-credential-socket\") pod \"3a0f28bc-d05f-45bb-8431-3aaf73529510\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") "
Nov 03 20:02:16.145782 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.145635    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-podinfo\") pod \"3a0f28bc-d05f-45bb-8431-3aaf73529510\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") "
Nov 03 20:02:16.145782 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.145704    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a0f28bc-d05f-45bb-8431-3aaf73529510-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a0f28bc-d05f-45bb-8431-3aaf73529510-istiod-ca-cert\") pod \"3a0f28bc-d05f-45bb-8431-3aaf73529510\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") "
Nov 03 20:02:16.145782 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.145772    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-workload-socket\") pod \"3a0f28bc-d05f-45bb-8431-3aaf73529510\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") "
Nov 03 20:02:16.146041 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.145797    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-envoy\") pod \"3a0f28bc-d05f-45bb-8431-3aaf73529510\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") "
Nov 03 20:02:16.146041 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.145976    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-workload-certs\") pod \"3a0f28bc-d05f-45bb-8431-3aaf73529510\" (UID: \"3a0f28bc-d05f-45bb-8431-3aaf73529510\") "
Nov 03 20:02:16.146643 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.145974    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-data" pod "3a0f28bc-d05f-45bb-8431-3aaf73529510" (UID: "3a0f28bc-d05f-45bb-8431-3aaf73529510"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:16.146829 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.146147    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-workload-socket" pod "3a0f28bc-d05f-45bb-8431-3aaf73529510" (UID: "3a0f28bc-d05f-45bb-8431-3aaf73529510"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:16.146924 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.146334    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-workload-certs" pod "3a0f28bc-d05f-45bb-8431-3aaf73529510" (UID: "3a0f28bc-d05f-45bb-8431-3aaf73529510"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:16.146993 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.146521    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a0f28bc-d05f-45bb-8431-3aaf73529510-istiod-ca-cert" pod "3a0f28bc-d05f-45bb-8431-3aaf73529510" (UID: "3a0f28bc-d05f-45bb-8431-3aaf73529510"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:16.147090 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.146646    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-credential-socket" pod "3a0f28bc-d05f-45bb-8431-3aaf73529510" (UID: "3a0f28bc-d05f-45bb-8431-3aaf73529510"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:16.147090 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.147042    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:16.148324 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.148300    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0f28bc-d05f-45bb-8431-3aaf73529510-kube-api-access-slwhg" pod "3a0f28bc-d05f-45bb-8431-3aaf73529510" (UID: "3a0f28bc-d05f-45bb-8431-3aaf73529510"). InnerVolumeSpecName "kube-api-access-slwhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:16.148568 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.148545    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-podinfo" pod "3a0f28bc-d05f-45bb-8431-3aaf73529510" (UID: "3a0f28bc-d05f-45bb-8431-3aaf73529510"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:02:16.148617 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.148572    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-token" pod "3a0f28bc-d05f-45bb-8431-3aaf73529510" (UID: "3a0f28bc-d05f-45bb-8431-3aaf73529510"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:16.148752 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.148738    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-envoy" pod "3a0f28bc-d05f-45bb-8431-3aaf73529510" (UID: "3a0f28bc-d05f-45bb-8431-3aaf73529510"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:16.217000 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.216962    5591 generic.go:356] "Generic (PLEG): container finished" podID="3a0f28bc-d05f-45bb-8431-3aaf73529510" containerID="00bc6fbf7920866477e22c63b8322d9015bf9059e8424f6465b6eb5fe6a77e8e" exitCode=0
Nov 03 20:02:16.217188 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.217100    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn" event={"ID":"3a0f28bc-d05f-45bb-8431-3aaf73529510","Type":"ContainerDied","Data":"00bc6fbf7920866477e22c63b8322d9015bf9059e8424f6465b6eb5fe6a77e8e"}
Nov 03 20:02:16.217188 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.217148    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn" event={"ID":"3a0f28bc-d05f-45bb-8431-3aaf73529510","Type":"ContainerDied","Data":"85cb93708fd1bb293bc411ecc1ea05defe98681b457b6e0e62104934ef4bc6a7"}
Nov 03 20:02:16.217188 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.217182    5591 scope.go:122] "RemoveContainer" containerID="00bc6fbf7920866477e22c63b8322d9015bf9059e8424f6465b6eb5fe6a77e8e"
Nov 03 20:02:16.217355 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.217337    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"
Nov 03 20:02:16.229311 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.229293    5591 scope.go:122] "RemoveContainer" containerID="00bc6fbf7920866477e22c63b8322d9015bf9059e8424f6465b6eb5fe6a77e8e"
Nov 03 20:02:16.229583 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:02:16.229568    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00bc6fbf7920866477e22c63b8322d9015bf9059e8424f6465b6eb5fe6a77e8e\": container with ID starting with 00bc6fbf7920866477e22c63b8322d9015bf9059e8424f6465b6eb5fe6a77e8e not found: ID does not exist" containerID="00bc6fbf7920866477e22c63b8322d9015bf9059e8424f6465b6eb5fe6a77e8e"
Nov 03 20:02:16.229636 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.229591    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00bc6fbf7920866477e22c63b8322d9015bf9059e8424f6465b6eb5fe6a77e8e"} err="failed to get container status \"00bc6fbf7920866477e22c63b8322d9015bf9059e8424f6465b6eb5fe6a77e8e\": rpc error: code = NotFound desc = could not find container \"00bc6fbf7920866477e22c63b8322d9015bf9059e8424f6465b6eb5fe6a77e8e\": container with ID starting with 00bc6fbf7920866477e22c63b8322d9015bf9059e8424f6465b6eb5fe6a77e8e not found: ID does not exist"
Nov 03 20:02:16.243246 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.243198    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"]
Nov 03 20:02:16.248372 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.248351    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-9cvzn"]
Nov 03 20:02:16.248893 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.248875    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:16.248944 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.248897    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slwhg\" (UniqueName: \"kubernetes.io/projected/3a0f28bc-d05f-45bb-8431-3aaf73529510-kube-api-access-slwhg\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:16.248944 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.248906    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:16.248944 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.248916    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:16.248944 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.248924    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:16.248944 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.248933    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a0f28bc-d05f-45bb-8431-3aaf73529510-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:16.248944 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.248941    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:16.249119 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:16.248954    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a0f28bc-d05f-45bb-8431-3aaf73529510-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:17.350295 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:17.350264    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a0f28bc-d05f-45bb-8431-3aaf73529510" path="/var/lib/kubelet/pods/3a0f28bc-d05f-45bb-8431-3aaf73529510/volumes"
Nov 03 20:02:21.790170 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:21.790138    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h"]
Nov 03 20:02:21.791043 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:21.790991    5591 cpu_manager.go:414] "containerMap: removing container" podUID="3a0f28bc-d05f-45bb-8431-3aaf73529510" containerName="istio-proxy"
Nov 03 20:02:21.791043 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:21.791017    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a0f28bc-d05f-45bb-8431-3aaf73529510" containerName="istio-proxy"
Nov 03 20:02:21.791323 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:21.791190    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a0f28bc-d05f-45bb-8431-3aaf73529510" containerName="istio-proxy"
Nov 03 20:02:21.796986 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:21.796953    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h"
Nov 03 20:02:21.818060 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:21.818026    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-c131fa\""
Nov 03 20:02:21.821858 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:21.821817    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h"]
Nov 03 20:02:21.836520 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:21.836485    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l"]
Nov 03 20:02:21.842727 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:21.842699    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l"
Nov 03 20:02:21.849289 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:21.849262    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l"]
Nov 03 20:02:21.859061 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:21.859017    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-c131fa-dockercfg-2mbmd\""
Nov 03 20:02:21.867140 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:21.867111    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp2r6\" (UniqueName: \"kubernetes.io/projected/221d9a79-e987-4b03-bd4f-e1f6718fe2ce-kube-api-access-fp2r6\") pod \"payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h\" (UID: \"221d9a79-e987-4b03-bd4f-e1f6718fe2ce\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h"
Nov 03 20:02:21.867308 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:21.867275    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221d9a79-e987-4b03-bd4f-e1f6718fe2ce-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h\" (UID: \"221d9a79-e987-4b03-bd4f-e1f6718fe2ce\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h"
Nov 03 20:02:21.976247 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:21.971278    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f72239c9-93c7-4f75-8296-3c8816841ac6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l\" (UID: \"f72239c9-93c7-4f75-8296-3c8816841ac6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l"
Nov 03 20:02:21.976247 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:21.971332    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjmz8\" (UniqueName: \"kubernetes.io/projected/f72239c9-93c7-4f75-8296-3c8816841ac6-kube-api-access-hjmz8\") pod \"payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l\" (UID: \"f72239c9-93c7-4f75-8296-3c8816841ac6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l"
Nov 03 20:02:21.976247 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:21.971423    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp2r6\" (UniqueName: \"kubernetes.io/projected/221d9a79-e987-4b03-bd4f-e1f6718fe2ce-kube-api-access-fp2r6\") pod \"payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h\" (UID: \"221d9a79-e987-4b03-bd4f-e1f6718fe2ce\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h"
Nov 03 20:02:21.976247 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:21.971521    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221d9a79-e987-4b03-bd4f-e1f6718fe2ce-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h\" (UID: \"221d9a79-e987-4b03-bd4f-e1f6718fe2ce\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h"
Nov 03 20:02:21.976247 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:21.972556    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221d9a79-e987-4b03-bd4f-e1f6718fe2ce-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h\" (UID: \"221d9a79-e987-4b03-bd4f-e1f6718fe2ce\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h"
Nov 03 20:02:22.037513 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:22.036024    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp2r6\" (UniqueName: \"kubernetes.io/projected/221d9a79-e987-4b03-bd4f-e1f6718fe2ce-kube-api-access-fp2r6\") pod \"payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h\" (UID: \"221d9a79-e987-4b03-bd4f-e1f6718fe2ce\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h"
Nov 03 20:02:22.079280 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:22.075438    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f72239c9-93c7-4f75-8296-3c8816841ac6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l\" (UID: \"f72239c9-93c7-4f75-8296-3c8816841ac6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l"
Nov 03 20:02:22.079280 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:22.075492    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjmz8\" (UniqueName: \"kubernetes.io/projected/f72239c9-93c7-4f75-8296-3c8816841ac6-kube-api-access-hjmz8\") pod \"payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l\" (UID: \"f72239c9-93c7-4f75-8296-3c8816841ac6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l"
Nov 03 20:02:22.079280 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:22.076760    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f72239c9-93c7-4f75-8296-3c8816841ac6-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l\" (UID: \"f72239c9-93c7-4f75-8296-3c8816841ac6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l"
Nov 03 20:02:22.114227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:22.107952    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjmz8\" (UniqueName: \"kubernetes.io/projected/f72239c9-93c7-4f75-8296-3c8816841ac6-kube-api-access-hjmz8\") pod \"payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l\" (UID: \"f72239c9-93c7-4f75-8296-3c8816841ac6\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l"
Nov 03 20:02:22.123253 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:22.116500    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h"
Nov 03 20:02:22.171943 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:22.171584    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l"
Nov 03 20:02:22.464705 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:22.464670    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h"]
Nov 03 20:02:22.466111 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:02:22.466079    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod221d9a79_e987_4b03_bd4f_e1f6718fe2ce.slice/crio-3ab2888d2a7574e5bb6a90b261a22efef4c7963046e067908141054538c1795c WatchSource:0}: Error finding container 3ab2888d2a7574e5bb6a90b261a22efef4c7963046e067908141054538c1795c: Status 404 returned error can't find the container with id 3ab2888d2a7574e5bb6a90b261a22efef4c7963046e067908141054538c1795c
Nov 03 20:02:22.488397 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:22.488375    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l"]
Nov 03 20:02:22.491557 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:02:22.491531    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf72239c9_93c7_4f75_8296_3c8816841ac6.slice/crio-154cebee71a013cde1b5a66a9d7e17cb8fc97046f8581057e4e8fc36e4164b4c WatchSource:0}: Error finding container 154cebee71a013cde1b5a66a9d7e17cb8fc97046f8581057e4e8fc36e4164b4c: Status 404 returned error can't find the container with id 154cebee71a013cde1b5a66a9d7e17cb8fc97046f8581057e4e8fc36e4164b4c
Nov 03 20:02:23.274331 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:23.274298    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l" event={"ID":"f72239c9-93c7-4f75-8296-3c8816841ac6","Type":"ContainerStarted","Data":"9228be36e0c2be328bb3f682348d2952b36d6ee8b7364822857923939e2c9831"}
Nov 03 20:02:23.274909 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:23.274359    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l" event={"ID":"f72239c9-93c7-4f75-8296-3c8816841ac6","Type":"ContainerStarted","Data":"154cebee71a013cde1b5a66a9d7e17cb8fc97046f8581057e4e8fc36e4164b4c"}
Nov 03 20:02:23.274909 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:23.274467    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l"
Nov 03 20:02:23.276187 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:23.276162    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h" event={"ID":"221d9a79-e987-4b03-bd4f-e1f6718fe2ce","Type":"ContainerStarted","Data":"7100a93dcba89dc90935f55ce5f558688fa1c946c18f95e6fb6241d170cc9998"}
Nov 03 20:02:23.276339 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:23.276250    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h" event={"ID":"221d9a79-e987-4b03-bd4f-e1f6718fe2ce","Type":"ContainerStarted","Data":"3ab2888d2a7574e5bb6a90b261a22efef4c7963046e067908141054538c1795c"}
Nov 03 20:02:23.276339 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:23.276282    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h"
Nov 03 20:02:23.294042 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:23.293995    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l" podStartSLOduration=2.293982895 podStartE2EDuration="2.293982895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:23.293155762 +0000 UTC m=+1492.065448780" watchObservedRunningTime="2026-11-03 20:02:23.293982895 +0000 UTC m=+1492.066275913"
Nov 03 20:02:23.315758 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:23.315712    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h" podStartSLOduration=2.3156997280000002 podStartE2EDuration="2.315699728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:23.312058229 +0000 UTC m=+1492.084351248" watchObservedRunningTime="2026-11-03 20:02:23.315699728 +0000 UTC m=+1492.087992775"
Nov 03 20:02:25.445082 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:25.445047    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4"]
Nov 03 20:02:25.449698 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:25.449667    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4"
Nov 03 20:02:25.461537 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:25.461105    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4"]
Nov 03 20:02:25.461537 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:25.461501    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-c131fa-serving-cert\""
Nov 03 20:02:25.550087 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:25.550058    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2tjk\" (UniqueName: \"kubernetes.io/projected/5f124fa7-5667-4c5d-a82f-352e57290fd6-kube-api-access-g2tjk\") pod \"maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4\" (UID: \"5f124fa7-5667-4c5d-a82f-352e57290fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4"
Nov 03 20:02:25.550311 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:25.550095    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f124fa7-5667-4c5d-a82f-352e57290fd6-metrics-certs\") pod \"maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4\" (UID: \"5f124fa7-5667-4c5d-a82f-352e57290fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4"
Nov 03 20:02:25.550398 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:25.550383    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f124fa7-5667-4c5d-a82f-352e57290fd6-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4\" (UID: \"5f124fa7-5667-4c5d-a82f-352e57290fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4"
Nov 03 20:02:25.550446 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:25.550430    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f124fa7-5667-4c5d-a82f-352e57290fd6-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4\" (UID: \"5f124fa7-5667-4c5d-a82f-352e57290fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4"
Nov 03 20:02:25.655234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:25.653991    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f124fa7-5667-4c5d-a82f-352e57290fd6-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4\" (UID: \"5f124fa7-5667-4c5d-a82f-352e57290fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4"
Nov 03 20:02:25.655234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:25.654086    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f124fa7-5667-4c5d-a82f-352e57290fd6-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4\" (UID: \"5f124fa7-5667-4c5d-a82f-352e57290fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4"
Nov 03 20:02:25.655234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:25.654152    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2tjk\" (UniqueName: \"kubernetes.io/projected/5f124fa7-5667-4c5d-a82f-352e57290fd6-kube-api-access-g2tjk\") pod \"maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4\" (UID: \"5f124fa7-5667-4c5d-a82f-352e57290fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4"
Nov 03 20:02:25.655234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:25.654188    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f124fa7-5667-4c5d-a82f-352e57290fd6-metrics-certs\") pod \"maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4\" (UID: \"5f124fa7-5667-4c5d-a82f-352e57290fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4"
Nov 03 20:02:25.655992 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:25.655959    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f124fa7-5667-4c5d-a82f-352e57290fd6-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4\" (UID: \"5f124fa7-5667-4c5d-a82f-352e57290fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4"
Nov 03 20:02:25.690552 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:25.687272    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f124fa7-5667-4c5d-a82f-352e57290fd6-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4\" (UID: \"5f124fa7-5667-4c5d-a82f-352e57290fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4"
Nov 03 20:02:25.690552 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:25.687950    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2tjk\" (UniqueName: \"kubernetes.io/projected/5f124fa7-5667-4c5d-a82f-352e57290fd6-kube-api-access-g2tjk\") pod \"maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4\" (UID: \"5f124fa7-5667-4c5d-a82f-352e57290fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4"
Nov 03 20:02:25.705152 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:25.703312    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f124fa7-5667-4c5d-a82f-352e57290fd6-metrics-certs\") pod \"maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4\" (UID: \"5f124fa7-5667-4c5d-a82f-352e57290fd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4"
Nov 03 20:02:25.777277 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:25.777248    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4"
Nov 03 20:02:26.167236 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:26.166493    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4"]
Nov 03 20:02:26.310846 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:26.310809    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4" event={"ID":"5f124fa7-5667-4c5d-a82f-352e57290fd6","Type":"ContainerStarted","Data":"a1b535d9a23fff2094dc160cb4040ef444baeacacee980657399d9a04205e0c8"}
Nov 03 20:02:27.410278 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:27.398351    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4" podStartSLOduration=1.8697735789999999 podStartE2EDuration="2.398330277s" totalImagesPullingTime="528.556698ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:27.39578837 +0000 UTC m=+1496.168081401" watchObservedRunningTime="2026-11-03 20:02:27.398330277 +0000 UTC m=+1496.170623297"
Nov 03 20:02:27.425232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:27.418044    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4"
Nov 03 20:02:27.425232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:27.418118    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4" event={"ID":"5f124fa7-5667-4c5d-a82f-352e57290fd6","Type":"ContainerStarted","Data":"5bc1db6a0147cbac984ed2ef48dfd65e91fed331bdfb1192c03eee9969d61848"}
Nov 03 20:02:28.633963 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:28.633316    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:28.889977 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:02:28.889844    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:02:29.077232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:29.076555    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:02:29.101260 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:29.100303    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:29.140407 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:29.140297    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:29.145227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:29.141804    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:29.166230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:29.165309    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:29.318230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:29.304315    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:29.454302 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:29.450307    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:29.639987 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:29.637750    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:29.639987 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:29.638419    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 20:02:29.915861 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:29.915819    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": context deadline exceeded"
Nov 03 20:02:30.108277 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:30.091465    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:30.123271 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:30.113766    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 20:02:30.225300 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:30.166105    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:02:30.225300 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:30.177877    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 20:02:30.269231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:30.106180    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:30.269231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:30.254359    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 20:02:30.370359 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:30.360391    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:30.370359 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:30.360960    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 20:02:30.468477 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:30.456607    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 20:02:30.468477 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:30.465492    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 20:02:30.683265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:30.670104    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:30.762967 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:30.762931    5591 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:02:30.790378 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:30.790303    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr" podUID="b043886c-68e1-48d8-9ec4-8ecd4601a753" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:31.101767 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:31.101709    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:31.173535 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:31.173489    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:02:31.190231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:31.184955    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:31.255439 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:31.255394    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:02:31.597498 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:31.597455    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 20:02:31.928281 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:31.912943    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded"
Nov 03 20:02:31.946275 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:31.938436    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 20:02:31.982364 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:31.982318    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:32.106349 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:32.106303    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:02:32.295286 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:32.241107    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:02:32.718908 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:32.718811    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:02:32.847467 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:32.847418    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 20:02:32.850481 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:32.850266    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:32.863131 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:32.863107    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 20:02:32.880459 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:32.880431    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 20:02:33.005416 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:33.005306    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 20:02:33.049667 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:33.037624    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 20:02:33.214779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:33.214749    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 20:02:33.254076 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:33.254038    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 20:02:33.421365 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:33.421334    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4"
Nov 03 20:02:33.427841 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:33.427817    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 20:02:33.428805 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:33.428777    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 20:02:34.302371 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:34.302338    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h"
Nov 03 20:02:34.303276 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:34.303255    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l"
Nov 03 20:02:40.912950 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:40.912866    5591 generic.go:356] "Generic (PLEG): container finished" podID="b12a508d-cf94-477f-8a7b-856231ac6aa5" containerID="431e5b0f265a02c8259b3be9f7f15edfc112cb2c0f3d46655f0b2cfd561be69a" exitCode=0
Nov 03 20:02:40.913380 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:40.912949    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght" event={"ID":"b12a508d-cf94-477f-8a7b-856231ac6aa5","Type":"ContainerDied","Data":"431e5b0f265a02c8259b3be9f7f15edfc112cb2c0f3d46655f0b2cfd561be69a"}
Nov 03 20:02:40.913445 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:40.913430    5591 scope.go:122] "RemoveContainer" containerID="431e5b0f265a02c8259b3be9f7f15edfc112cb2c0f3d46655f0b2cfd561be69a"
Nov 03 20:02:40.914941 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:40.914914    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:02:40.915021 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:40.914998    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:02:40.915056 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:40.915030    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:02:41.921162 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:41.921127    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght" event={"ID":"b12a508d-cf94-477f-8a7b-856231ac6aa5","Type":"ContainerStarted","Data":"b2c874104a14463063f957077b665b15cd9dc02123533f91fed166aa36db1506"}
Nov 03 20:02:42.380558 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:42.380528    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:02:42.381830 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:42.381792    5591 patch_prober.go:29] interesting pod/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 03 20:02:42.381980 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:42.381854    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght" podUID="b12a508d-cf94-477f-8a7b-856231ac6aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 03 20:02:43.402226 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:43.397616    5591 patch_prober.go:29] interesting pod/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 03 20:02:43.402226 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:43.397680    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght" podUID="b12a508d-cf94-477f-8a7b-856231ac6aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 03 20:02:44.043899 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:44.043863    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:02:44.384286 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:44.382977    5591 patch_prober.go:29] interesting pod/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 03 20:02:44.384286 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:44.383043    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght" podUID="b12a508d-cf94-477f-8a7b-856231ac6aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 03 20:02:45.396678 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:45.396638    5591 patch_prober.go:29] interesting pod/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 03 20:02:45.397321 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:45.397289    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght" podUID="b12a508d-cf94-477f-8a7b-856231ac6aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 03 20:02:45.444370 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:45.444329    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:02:46.384227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:46.383693    5591 patch_prober.go:29] interesting pod/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 03 20:02:46.384227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:46.383751    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght" podUID="b12a508d-cf94-477f-8a7b-856231ac6aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 03 20:02:47.392354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:47.392316    5591 patch_prober.go:29] interesting pod/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 03 20:02:47.392936 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:47.392906    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght" podUID="b12a508d-cf94-477f-8a7b-856231ac6aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 03 20:02:47.870359 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:47.870311    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:02:48.076232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:48.075358    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": context deadline exceeded"
Nov 03 20:02:48.154426 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:48.154328    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:48.255428 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:48.255377    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 20:02:48.302359 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:48.302302    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:48.386660 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:48.386619    5591 patch_prober.go:29] interesting pod/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 03 20:02:48.387017 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:48.386983    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght" podUID="b12a508d-cf94-477f-8a7b-856231ac6aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 03 20:02:48.615340 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:48.615295    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:48.705388 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:48.704326    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:48.770924 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:48.770879    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd"]
Nov 03 20:02:48.851227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:48.850587    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts"]
Nov 03 20:02:48.871817 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:48.871717    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:48.896274 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:48.878291    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:48.896274 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:48.878712    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 20:02:49.177631 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:49.164633    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:02:49.177631 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:49.167698    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 20:02:49.362231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:49.341061    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded"
Nov 03 20:02:49.497279 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:49.393551    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:49.694149 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:49.546890    5591 patch_prober.go:29] interesting pod/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 03 20:02:49.706466 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:49.706419    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght" podUID="b12a508d-cf94-477f-8a7b-856231ac6aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 03 20:02:49.752132 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:49.727428    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 20:02:49.753466 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:49.393728    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 20:02:49.816724 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:49.734453    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:50.102233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:49.820178    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:50.102233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:49.862150    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 20:02:50.102233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:50.090818    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 20:02:50.102233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:50.090886    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 20:02:50.102233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:50.090940    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 20:02:50.102233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:49.881072    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:50.103522 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:49.881115    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:50.105898 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:50.105874    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 20:02:50.276547 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:50.276499    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:02:50.564288 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:50.476557    5591 patch_prober.go:29] interesting pod/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 03 20:02:50.564288 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:50.476618    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght" podUID="b12a508d-cf94-477f-8a7b-856231ac6aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 03 20:02:50.564288 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:50.510953    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:50.595231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:50.570511    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"]
Nov 03 20:02:50.595856 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:50.595807    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq" podUID="3c9a677a-d699-4b95-b217-09c48c89dff5" containerName="istio-proxy" containerID="cri-o://53f6267c02996dfe84da999d23e8a68e263db947788f32841483738cff87eddd" gracePeriod=30
Nov 03 20:02:50.757898 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:50.757845    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 20:02:50.758948 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:50.758907    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 20:02:50.833289 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:50.829187    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:51.109443 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:51.109350    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:51.115709 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:51.109707    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:51.381574 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:51.381475    5591 patch_prober.go:29] interesting pod/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 03 20:02:51.381886 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:51.381857    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght" podUID="b12a508d-cf94-477f-8a7b-856231ac6aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 03 20:02:51.386453 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:51.386405    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:51.573259 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:51.573222    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 20:02:51.603102 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:51.603058    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded"
Nov 03 20:02:51.767301 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:51.758970    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 20:02:51.987127 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:51.922590    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:51.987341 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:51.922628    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 20:02:52.332295 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:52.188360    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:52.332295 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:52.321955    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:52.419824 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:52.404335    5591 patch_prober.go:29] interesting pod/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 03 20:02:52.420238 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:52.420187    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght" podUID="b12a508d-cf94-477f-8a7b-856231ac6aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 03 20:02:52.455238 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:02:52.436739    5591 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.043s"
Nov 03 20:02:52.458534 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:52.458509    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 20:02:52.618293 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:52.600093    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:52.830693 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:52.830500    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 20:02:52.835163 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:52.835100    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 20:02:52.859706 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:52.852840    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 20:02:52.881870 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:52.881785    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 20:02:52.882870 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:52.882847    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 20:02:52.897415 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:52.897390    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 20:02:53.406028 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:53.405994    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:02:53.427023 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:53.426990    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:02:53.772040 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:53.772001    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd" podUID="1837d3e9-0dc6-4e35-b915-688a59b244bc" containerName="payload-processing" containerID="cri-o://9d4ad3129312dabba2f5b46d5237436340299c242b8d02c59b2d6aa850f879c2" gracePeriod=25
Nov 03 20:02:53.851896 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:53.851857    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts" podUID="bb7b5608-93ed-41f2-a0ed-995d6041bf70" containerName="payload-pre-processing" containerID="cri-o://70cad7d7a9903c5b58e8e2dad32cc24d8ab36d3ddfe233f24f85760124b00385" gracePeriod=25
Nov 03 20:02:54.147290 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.147262    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd"
Nov 03 20:02:54.216919 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.216894    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts"
Nov 03 20:02:54.285699 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.285608    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1837d3e9-0dc6-4e35-b915-688a59b244bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1837d3e9-0dc6-4e35-b915-688a59b244bc-plugins-config-volume\") pod \"1837d3e9-0dc6-4e35-b915-688a59b244bc\" (UID: \"1837d3e9-0dc6-4e35-b915-688a59b244bc\") "
Nov 03 20:02:54.285872 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.285759    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb7b5608-93ed-41f2-a0ed-995d6041bf70-kube-api-access-zzxbh\" (UniqueName: \"kubernetes.io/projected/bb7b5608-93ed-41f2-a0ed-995d6041bf70-kube-api-access-zzxbh\") pod \"bb7b5608-93ed-41f2-a0ed-995d6041bf70\" (UID: \"bb7b5608-93ed-41f2-a0ed-995d6041bf70\") "
Nov 03 20:02:54.285934 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.285914    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1837d3e9-0dc6-4e35-b915-688a59b244bc-kube-api-access-fbkmc\" (UniqueName: \"kubernetes.io/projected/1837d3e9-0dc6-4e35-b915-688a59b244bc-kube-api-access-fbkmc\") pod \"1837d3e9-0dc6-4e35-b915-688a59b244bc\" (UID: \"1837d3e9-0dc6-4e35-b915-688a59b244bc\") "
Nov 03 20:02:54.286018 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.286001    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb7b5608-93ed-41f2-a0ed-995d6041bf70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7b5608-93ed-41f2-a0ed-995d6041bf70-plugins-config-volume\") pod \"bb7b5608-93ed-41f2-a0ed-995d6041bf70\" (UID: \"bb7b5608-93ed-41f2-a0ed-995d6041bf70\") "
Nov 03 20:02:54.286089 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.286045    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1837d3e9-0dc6-4e35-b915-688a59b244bc-plugins-config-volume" pod "1837d3e9-0dc6-4e35-b915-688a59b244bc" (UID: "1837d3e9-0dc6-4e35-b915-688a59b244bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:54.286563 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.286535    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb7b5608-93ed-41f2-a0ed-995d6041bf70-plugins-config-volume" pod "bb7b5608-93ed-41f2-a0ed-995d6041bf70" (UID: "bb7b5608-93ed-41f2-a0ed-995d6041bf70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:54.286701 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.286597    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1837d3e9-0dc6-4e35-b915-688a59b244bc-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:54.288125 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.288098    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7b5608-93ed-41f2-a0ed-995d6041bf70-kube-api-access-zzxbh" pod "bb7b5608-93ed-41f2-a0ed-995d6041bf70" (UID: "bb7b5608-93ed-41f2-a0ed-995d6041bf70"). InnerVolumeSpecName "kube-api-access-zzxbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:54.288301 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.288280    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1837d3e9-0dc6-4e35-b915-688a59b244bc-kube-api-access-fbkmc" pod "1837d3e9-0dc6-4e35-b915-688a59b244bc" (UID: "1837d3e9-0dc6-4e35-b915-688a59b244bc"). InnerVolumeSpecName "kube-api-access-fbkmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:54.387983 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.387949    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzxbh\" (UniqueName: \"kubernetes.io/projected/bb7b5608-93ed-41f2-a0ed-995d6041bf70-kube-api-access-zzxbh\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:54.387983 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.387983    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbkmc\" (UniqueName: \"kubernetes.io/projected/1837d3e9-0dc6-4e35-b915-688a59b244bc-kube-api-access-fbkmc\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:54.388181 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.387995    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb7b5608-93ed-41f2-a0ed-995d6041bf70-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:54.490885 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.490847    5591 generic.go:356] "Generic (PLEG): container finished" podID="bb7b5608-93ed-41f2-a0ed-995d6041bf70" containerID="70cad7d7a9903c5b58e8e2dad32cc24d8ab36d3ddfe233f24f85760124b00385" exitCode=0
Nov 03 20:02:54.491095 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.490904    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts"
Nov 03 20:02:54.491095 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.490936    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts" event={"ID":"bb7b5608-93ed-41f2-a0ed-995d6041bf70","Type":"ContainerDied","Data":"70cad7d7a9903c5b58e8e2dad32cc24d8ab36d3ddfe233f24f85760124b00385"}
Nov 03 20:02:54.491095 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.490992    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts" event={"ID":"bb7b5608-93ed-41f2-a0ed-995d6041bf70","Type":"ContainerDied","Data":"4ec7e5984fb254b2baec03150a7e549f4d8baa69fcb9a5ece787e536a5f2ad9e"}
Nov 03 20:02:54.491095 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.491022    5591 scope.go:122] "RemoveContainer" containerID="70cad7d7a9903c5b58e8e2dad32cc24d8ab36d3ddfe233f24f85760124b00385"
Nov 03 20:02:54.492970 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.492939    5591 generic.go:356] "Generic (PLEG): container finished" podID="1837d3e9-0dc6-4e35-b915-688a59b244bc" containerID="9d4ad3129312dabba2f5b46d5237436340299c242b8d02c59b2d6aa850f879c2" exitCode=0
Nov 03 20:02:54.493112 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.492996    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd" event={"ID":"1837d3e9-0dc6-4e35-b915-688a59b244bc","Type":"ContainerDied","Data":"9d4ad3129312dabba2f5b46d5237436340299c242b8d02c59b2d6aa850f879c2"}
Nov 03 20:02:54.493112 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.493042    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd" event={"ID":"1837d3e9-0dc6-4e35-b915-688a59b244bc","Type":"ContainerDied","Data":"29f1652ea596e833514e3ecc2ce42681f20c0266add3a7772f30dd60abfdf244"}
Nov 03 20:02:54.493223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.493152    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd"
Nov 03 20:02:54.503510 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.503488    5591 scope.go:122] "RemoveContainer" containerID="70cad7d7a9903c5b58e8e2dad32cc24d8ab36d3ddfe233f24f85760124b00385"
Nov 03 20:02:54.503829 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:02:54.503788    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70cad7d7a9903c5b58e8e2dad32cc24d8ab36d3ddfe233f24f85760124b00385\": container with ID starting with 70cad7d7a9903c5b58e8e2dad32cc24d8ab36d3ddfe233f24f85760124b00385 not found: ID does not exist" containerID="70cad7d7a9903c5b58e8e2dad32cc24d8ab36d3ddfe233f24f85760124b00385"
Nov 03 20:02:54.503829 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.503819    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70cad7d7a9903c5b58e8e2dad32cc24d8ab36d3ddfe233f24f85760124b00385"} err="failed to get container status \"70cad7d7a9903c5b58e8e2dad32cc24d8ab36d3ddfe233f24f85760124b00385\": rpc error: code = NotFound desc = could not find container \"70cad7d7a9903c5b58e8e2dad32cc24d8ab36d3ddfe233f24f85760124b00385\": container with ID starting with 70cad7d7a9903c5b58e8e2dad32cc24d8ab36d3ddfe233f24f85760124b00385 not found: ID does not exist"
Nov 03 20:02:54.503944 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.503837    5591 scope.go:122] "RemoveContainer" containerID="9d4ad3129312dabba2f5b46d5237436340299c242b8d02c59b2d6aa850f879c2"
Nov 03 20:02:54.513318 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.513191    5591 scope.go:122] "RemoveContainer" containerID="9d4ad3129312dabba2f5b46d5237436340299c242b8d02c59b2d6aa850f879c2"
Nov 03 20:02:54.513519 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:02:54.513501    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d4ad3129312dabba2f5b46d5237436340299c242b8d02c59b2d6aa850f879c2\": container with ID starting with 9d4ad3129312dabba2f5b46d5237436340299c242b8d02c59b2d6aa850f879c2 not found: ID does not exist" containerID="9d4ad3129312dabba2f5b46d5237436340299c242b8d02c59b2d6aa850f879c2"
Nov 03 20:02:54.513568 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.513525    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d4ad3129312dabba2f5b46d5237436340299c242b8d02c59b2d6aa850f879c2"} err="failed to get container status \"9d4ad3129312dabba2f5b46d5237436340299c242b8d02c59b2d6aa850f879c2\": rpc error: code = NotFound desc = could not find container \"9d4ad3129312dabba2f5b46d5237436340299c242b8d02c59b2d6aa850f879c2\": container with ID starting with 9d4ad3129312dabba2f5b46d5237436340299c242b8d02c59b2d6aa850f879c2 not found: ID does not exist"
Nov 03 20:02:54.519690 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.519667    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts"]
Nov 03 20:02:54.526901 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.526882    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8fa615-54c4d8547f-45cts"]
Nov 03 20:02:54.539117 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.539056    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd"]
Nov 03 20:02:54.543278 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:54.543255    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8fa615-54b7cc8fb6-rgqgd"]
Nov 03 20:02:55.350351 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:55.350318    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1837d3e9-0dc6-4e35-b915-688a59b244bc" path="/var/lib/kubelet/pods/1837d3e9-0dc6-4e35-b915-688a59b244bc/volumes"
Nov 03 20:02:55.350807 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:55.350793    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb7b5608-93ed-41f2-a0ed-995d6041bf70" path="/var/lib/kubelet/pods/bb7b5608-93ed-41f2-a0ed-995d6041bf70/volumes"
Nov 03 20:02:56.449704 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.449666    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"]
Nov 03 20:02:56.450527 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.450505    5591 cpu_manager.go:414] "containerMap: removing container" podUID="1837d3e9-0dc6-4e35-b915-688a59b244bc" containerName="payload-processing"
Nov 03 20:02:56.450527 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.450525    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1837d3e9-0dc6-4e35-b915-688a59b244bc" containerName="payload-processing"
Nov 03 20:02:56.450680 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.450536    5591 cpu_manager.go:414] "containerMap: removing container" podUID="bb7b5608-93ed-41f2-a0ed-995d6041bf70" containerName="payload-pre-processing"
Nov 03 20:02:56.450680 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.450541    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb7b5608-93ed-41f2-a0ed-995d6041bf70" containerName="payload-pre-processing"
Nov 03 20:02:56.450680 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.450674    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="1837d3e9-0dc6-4e35-b915-688a59b244bc" containerName="payload-processing"
Nov 03 20:02:56.450779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.450681    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb7b5608-93ed-41f2-a0ed-995d6041bf70" containerName="payload-pre-processing"
Nov 03 20:02:56.462981 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.462950    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:56.472977 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.472947    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-4fb1d0-openshift-default-dockercfg-tqkqh\""
Nov 03 20:02:56.481074 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.481040    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"]
Nov 03 20:02:56.639023 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.638989    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-credential-socket\") pod \"e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") " pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:56.639307 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.639050    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-envoy\") pod \"e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") " pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:56.639307 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.639252    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-workload-certs\") pod \"e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") " pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:56.639449 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.639339    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-token\") pod \"e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") " pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:56.639449 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.639368    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qdn5\" (UniqueName: \"kubernetes.io/projected/7aff0e54-0f19-41b1-af95-70aee78fea74-kube-api-access-5qdn5\") pod \"e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") " pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:56.639449 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.639414    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aff0e54-0f19-41b1-af95-70aee78fea74-istiod-ca-cert\") pod \"e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") " pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:56.639624 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.639555    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-workload-socket\") pod \"e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") " pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:56.639624 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.639604    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-data\") pod \"e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") " pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:56.639770 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.639755    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-podinfo\") pod \"e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") " pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:56.741506 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.741407    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-envoy\") pod \"e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") " pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:56.748403 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.748368    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-workload-certs\") pod \"e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") " pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:56.748677 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.748657    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-token\") pod \"e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") " pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:56.748816 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.748798    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qdn5\" (UniqueName: \"kubernetes.io/projected/7aff0e54-0f19-41b1-af95-70aee78fea74-kube-api-access-5qdn5\") pod \"e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") " pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:56.749012 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.748999    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aff0e54-0f19-41b1-af95-70aee78fea74-istiod-ca-cert\") pod \"e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") " pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:56.749143 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.749127    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-workload-socket\") pod \"e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") " pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:56.749316 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.749300    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-data\") pod \"e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") " pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:56.749553 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.749531    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-podinfo\") pod \"e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") " pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:56.757802 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.749769    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-credential-socket\") pod \"e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") " pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:56.758680 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.758651    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-credential-socket\") pod \"e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") " pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:56.761408 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.759396    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aff0e54-0f19-41b1-af95-70aee78fea74-istiod-ca-cert\") pod \"e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") " pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:56.761593 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.759650    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-workload-certs\") pod \"e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") " pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:56.761713 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.760324    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-data\") pod \"e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") " pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:56.761816 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.760505    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-workload-socket\") pod \"e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") " pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:56.785671 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.785598    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-podinfo\") pod \"e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") " pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:56.792819 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.792787    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-envoy\") pod \"e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") " pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:56.796348 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.796304    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-token\") pod \"e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") " pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:56.811264 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:56.811219    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qdn5\" (UniqueName: \"kubernetes.io/projected/7aff0e54-0f19-41b1-af95-70aee78fea74-kube-api-access-5qdn5\") pod \"e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") " pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:57.092971 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.092938    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:57.331495 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.331457    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf"]
Nov 03 20:02:57.335801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.335777    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf"
Nov 03 20:02:57.359174 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.359139    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf"]
Nov 03 20:02:57.360071 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.360043    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-942930\""
Nov 03 20:02:57.370888 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.370758    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993ba6c4-13eb-4630-870c-e6018b883996-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf\" (UID: \"993ba6c4-13eb-4630-870c-e6018b883996\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf"
Nov 03 20:02:57.371799 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.371760    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9rn7\" (UniqueName: \"kubernetes.io/projected/993ba6c4-13eb-4630-870c-e6018b883996-kube-api-access-r9rn7\") pod \"payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf\" (UID: \"993ba6c4-13eb-4630-870c-e6018b883996\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf"
Nov 03 20:02:57.395034 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.395001    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-942930-dockercfg-sqnq5\""
Nov 03 20:02:57.400831 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:02:57.400782    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aff0e54_0f19_41b1_af95_70aee78fea74.slice/crio-e12a74a306289aea3aa9f7d44645922676a293a5a8966863925a957373ce3af2 WatchSource:0}: Error finding container e12a74a306289aea3aa9f7d44645922676a293a5a8966863925a957373ce3af2: Status 404 returned error can't find the container with id e12a74a306289aea3aa9f7d44645922676a293a5a8966863925a957373ce3af2
Nov 03 20:02:57.414686 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.414658    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl"]
Nov 03 20:02:57.418891 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.418842    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:02:57.419047 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.418931    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:02:57.419047 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.418974    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:02:57.455313 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.452780    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"]
Nov 03 20:02:57.455313 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.452931    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl"
Nov 03 20:02:57.574659 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.481534    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j8zb\" (UniqueName: \"kubernetes.io/projected/c1b5af35-9a4e-4820-ba02-3c5efb129d45-kube-api-access-7j8zb\") pod \"payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl\" (UID: \"c1b5af35-9a4e-4820-ba02-3c5efb129d45\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl"
Nov 03 20:02:57.574659 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.481629    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b5af35-9a4e-4820-ba02-3c5efb129d45-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl\" (UID: \"c1b5af35-9a4e-4820-ba02-3c5efb129d45\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl"
Nov 03 20:02:57.574659 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.481667    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993ba6c4-13eb-4630-870c-e6018b883996-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf\" (UID: \"993ba6c4-13eb-4630-870c-e6018b883996\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf"
Nov 03 20:02:57.574659 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.481726    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9rn7\" (UniqueName: \"kubernetes.io/projected/993ba6c4-13eb-4630-870c-e6018b883996-kube-api-access-r9rn7\") pod \"payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf\" (UID: \"993ba6c4-13eb-4630-870c-e6018b883996\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf"
Nov 03 20:02:57.574659 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.483115    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993ba6c4-13eb-4630-870c-e6018b883996-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf\" (UID: \"993ba6c4-13eb-4630-870c-e6018b883996\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf"
Nov 03 20:02:57.619159 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.619007    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9rn7\" (UniqueName: \"kubernetes.io/projected/993ba6c4-13eb-4630-870c-e6018b883996-kube-api-access-r9rn7\") pod \"payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf\" (UID: \"993ba6c4-13eb-4630-870c-e6018b883996\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf"
Nov 03 20:02:57.659980 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.650815    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j8zb\" (UniqueName: \"kubernetes.io/projected/c1b5af35-9a4e-4820-ba02-3c5efb129d45-kube-api-access-7j8zb\") pod \"payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl\" (UID: \"c1b5af35-9a4e-4820-ba02-3c5efb129d45\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl"
Nov 03 20:02:57.659980 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.650912    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b5af35-9a4e-4820-ba02-3c5efb129d45-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl\" (UID: \"c1b5af35-9a4e-4820-ba02-3c5efb129d45\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl"
Nov 03 20:02:57.659980 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.651938    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b5af35-9a4e-4820-ba02-3c5efb129d45-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl\" (UID: \"c1b5af35-9a4e-4820-ba02-3c5efb129d45\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl"
Nov 03 20:02:57.659980 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.658272    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl"]
Nov 03 20:02:57.663771 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.663572    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf"
Nov 03 20:02:57.700080 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.700023    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j8zb\" (UniqueName: \"kubernetes.io/projected/c1b5af35-9a4e-4820-ba02-3c5efb129d45-kube-api-access-7j8zb\") pod \"payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl\" (UID: \"c1b5af35-9a4e-4820-ba02-3c5efb129d45\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl"
Nov 03 20:02:57.705779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.705677    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f" event={"ID":"7aff0e54-0f19-41b1-af95-70aee78fea74","Type":"ContainerStarted","Data":"e12a74a306289aea3aa9f7d44645922676a293a5a8966863925a957373ce3af2"}
Nov 03 20:02:57.706752 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.706491    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 20:02:57.707989 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.707962    5591 generic.go:356] "Generic (PLEG): container finished" podID="3c9a677a-d699-4b95-b217-09c48c89dff5" containerID="53f6267c02996dfe84da999d23e8a68e263db947788f32841483738cff87eddd" exitCode=0
Nov 03 20:02:57.708141 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.708122    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq" event={"ID":"3c9a677a-d699-4b95-b217-09c48c89dff5","Type":"ContainerDied","Data":"53f6267c02996dfe84da999d23e8a68e263db947788f32841483738cff87eddd"}
Nov 03 20:02:57.708290 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.708181    5591 scope.go:122] "RemoveContainer" containerID="53f6267c02996dfe84da999d23e8a68e263db947788f32841483738cff87eddd"
Nov 03 20:02:57.758124 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.752103    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c9a677a-d699-4b95-b217-09c48c89dff5-kube-api-access-mphk8\" (UniqueName: \"kubernetes.io/projected/3c9a677a-d699-4b95-b217-09c48c89dff5-kube-api-access-mphk8\") pod \"3c9a677a-d699-4b95-b217-09c48c89dff5\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") "
Nov 03 20:02:57.758124 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.752175    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-data\") pod \"3c9a677a-d699-4b95-b217-09c48c89dff5\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") "
Nov 03 20:02:57.758124 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.752230    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-workload-certs\") pod \"3c9a677a-d699-4b95-b217-09c48c89dff5\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") "
Nov 03 20:02:57.758124 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.752325    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-credential-socket\") pod \"3c9a677a-d699-4b95-b217-09c48c89dff5\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") "
Nov 03 20:02:57.758124 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.752367    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-workload-socket\") pod \"3c9a677a-d699-4b95-b217-09c48c89dff5\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") "
Nov 03 20:02:57.758124 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.752421    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-podinfo\") pod \"3c9a677a-d699-4b95-b217-09c48c89dff5\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") "
Nov 03 20:02:57.758124 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.752486    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-envoy\") pod \"3c9a677a-d699-4b95-b217-09c48c89dff5\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") "
Nov 03 20:02:57.758124 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.752573    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c9a677a-d699-4b95-b217-09c48c89dff5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c9a677a-d699-4b95-b217-09c48c89dff5-istiod-ca-cert\") pod \"3c9a677a-d699-4b95-b217-09c48c89dff5\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") "
Nov 03 20:02:57.758124 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.752621    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-token\" (UniqueName: \"kubernetes.io/projected/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-token\") pod \"3c9a677a-d699-4b95-b217-09c48c89dff5\" (UID: \"3c9a677a-d699-4b95-b217-09c48c89dff5\") "
Nov 03 20:02:57.758124 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.755003    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-credential-socket" pod "3c9a677a-d699-4b95-b217-09c48c89dff5" (UID: "3c9a677a-d699-4b95-b217-09c48c89dff5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:57.758124 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.756852    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-data" pod "3c9a677a-d699-4b95-b217-09c48c89dff5" (UID: "3c9a677a-d699-4b95-b217-09c48c89dff5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:57.758124 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.757081    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-workload-socket" pod "3c9a677a-d699-4b95-b217-09c48c89dff5" (UID: "3c9a677a-d699-4b95-b217-09c48c89dff5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:57.758124 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.757405    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c9a677a-d699-4b95-b217-09c48c89dff5-istiod-ca-cert" pod "3c9a677a-d699-4b95-b217-09c48c89dff5" (UID: "3c9a677a-d699-4b95-b217-09c48c89dff5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:57.758124 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.757645    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-workload-certs" pod "3c9a677a-d699-4b95-b217-09c48c89dff5" (UID: "3c9a677a-d699-4b95-b217-09c48c89dff5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:57.782166 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.782097    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-envoy" pod "3c9a677a-d699-4b95-b217-09c48c89dff5" (UID: "3c9a677a-d699-4b95-b217-09c48c89dff5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:02:57.783296 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.782477    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-podinfo" pod "3c9a677a-d699-4b95-b217-09c48c89dff5" (UID: "3c9a677a-d699-4b95-b217-09c48c89dff5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:02:57.783812 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.783773    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-token" pod "3c9a677a-d699-4b95-b217-09c48c89dff5" (UID: "3c9a677a-d699-4b95-b217-09c48c89dff5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:57.787536 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.787497    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9a677a-d699-4b95-b217-09c48c89dff5-kube-api-access-mphk8" pod "3c9a677a-d699-4b95-b217-09c48c89dff5" (UID: "3c9a677a-d699-4b95-b217-09c48c89dff5"). InnerVolumeSpecName "kube-api-access-mphk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:57.857828 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.857671    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:57.857828 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.857704    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:57.857828 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.857718    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:57.857828 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.857736    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:57.857828 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.857750    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c9a677a-d699-4b95-b217-09c48c89dff5-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:57.857828 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.857764    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:57.857828 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.857778    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mphk8\" (UniqueName: \"kubernetes.io/projected/3c9a677a-d699-4b95-b217-09c48c89dff5-kube-api-access-mphk8\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:57.857828 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.857792    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:57.857828 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.857805    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c9a677a-d699-4b95-b217-09c48c89dff5-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:57.861196 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:57.860952    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl"
Nov 03 20:02:58.070774 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:58.070119    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf"]
Nov 03 20:02:58.257501 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:58.257475    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl"]
Nov 03 20:02:58.258784 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:02:58.258750    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b5af35_9a4e_4820_ba02_3c5efb129d45.slice/crio-e42de72e74a8395660316d326b032d2cd5ff34971732a87574cb3dd5271a400f WatchSource:0}: Error finding container e42de72e74a8395660316d326b032d2cd5ff34971732a87574cb3dd5271a400f: Status 404 returned error can't find the container with id e42de72e74a8395660316d326b032d2cd5ff34971732a87574cb3dd5271a400f
Nov 03 20:02:58.418734 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:58.417471    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c"]
Nov 03 20:02:58.418734 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:58.417696    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c" podUID="1a0b68e2-5f67-415a-96d0-8186134792bc" containerName="maas-api" containerID="cri-o://8b987c3f3c8ee227f46723ca2c8809b1b8f59b99eeb21393b39233e165192723" gracePeriod=30
Nov 03 20:02:58.455540 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:58.455499    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:02:58.456009 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:58.455570    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq" podUID="3c9a677a-d699-4b95-b217-09c48c89dff5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:02:58.891318 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:58.890168    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f" event={"ID":"7aff0e54-0f19-41b1-af95-70aee78fea74","Type":"ContainerStarted","Data":"519e0c1283e7fc13fbf46f1e602664f635ae39749f0aae6bbb1f164e4a7f884f"}
Nov 03 20:02:58.892897 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:58.892868    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq" event={"ID":"3c9a677a-d699-4b95-b217-09c48c89dff5","Type":"ContainerDied","Data":"2ae6c2be53e5d90f394ac30a7bc90ca99908a4740aac0ea3c7a99391a20c2343"}
Nov 03 20:02:58.893165 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:58.893150    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"
Nov 03 20:02:58.909235 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:58.909034    5591 generic.go:356] "Generic (PLEG): container finished" podID="1a0b68e2-5f67-415a-96d0-8186134792bc" containerID="8b987c3f3c8ee227f46723ca2c8809b1b8f59b99eeb21393b39233e165192723" exitCode=0
Nov 03 20:02:58.909235 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:58.909184    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c" event={"ID":"1a0b68e2-5f67-415a-96d0-8186134792bc","Type":"ContainerDied","Data":"8b987c3f3c8ee227f46723ca2c8809b1b8f59b99eeb21393b39233e165192723"}
Nov 03 20:02:58.929534 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:58.918231    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl" event={"ID":"c1b5af35-9a4e-4820-ba02-3c5efb129d45","Type":"ContainerStarted","Data":"5f49f4b7a7e1738aca0c1a85f3a78b4481a97ed7c6eeacdd2dc97f3e6f24280a"}
Nov 03 20:02:58.929534 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:58.918298    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl" event={"ID":"c1b5af35-9a4e-4820-ba02-3c5efb129d45","Type":"ContainerStarted","Data":"e42de72e74a8395660316d326b032d2cd5ff34971732a87574cb3dd5271a400f"}
Nov 03 20:02:58.929534 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:58.919282    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl"
Nov 03 20:02:59.014342 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:59.014288    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f" podStartSLOduration=3.014266685 podStartE2EDuration="3.014266685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:58.952851221 +0000 UTC m=+1527.725144240" watchObservedRunningTime="2026-11-03 20:02:59.014266685 +0000 UTC m=+1527.786559713"
Nov 03 20:02:59.024246 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:02:59.024131    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:02:59.025850 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:59.024839    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf" event={"ID":"993ba6c4-13eb-4630-870c-e6018b883996","Type":"ContainerStarted","Data":"76490e2959f68b71475feea24ce66ba9c1134d31b281dca1014e6c087cdc25a0"}
Nov 03 20:02:59.025850 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:59.024892    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf" event={"ID":"993ba6c4-13eb-4630-870c-e6018b883996","Type":"ContainerStarted","Data":"529f98f598dd5996432ec5711b9fd662865c244b0d28124c7dac1692e039f045"}
Nov 03 20:02:59.025850 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:59.025796    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf"
Nov 03 20:02:59.027270 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:59.026887    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl" podStartSLOduration=2.026870336 podStartE2EDuration="2.026870336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:59.01249608 +0000 UTC m=+1527.784789099" watchObservedRunningTime="2026-11-03 20:02:59.026870336 +0000 UTC m=+1527.799163355"
Nov 03 20:02:59.031967 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:59.031941    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c"
Nov 03 20:02:59.034662 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:59.034640    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"]
Nov 03 20:02:59.039064 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:59.039041    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-bk5wq"]
Nov 03 20:02:59.055133 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:59.055089    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf" podStartSLOduration=2.05507657 podStartE2EDuration="2.05507657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:02:59.052409132 +0000 UTC m=+1527.824702155" watchObservedRunningTime="2026-11-03 20:02:59.05507657 +0000 UTC m=+1527.827369579"
Nov 03 20:02:59.093723 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:59.093690    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:59.098822 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:59.098796    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:02:59.152256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:59.152168    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a0b68e2-5f67-415a-96d0-8186134792bc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a0b68e2-5f67-415a-96d0-8186134792bc-openshift-service-ca\") pod \"1a0b68e2-5f67-415a-96d0-8186134792bc\" (UID: \"1a0b68e2-5f67-415a-96d0-8186134792bc\") "
Nov 03 20:02:59.152402 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:59.152300    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a0b68e2-5f67-415a-96d0-8186134792bc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a0b68e2-5f67-415a-96d0-8186134792bc-metrics-certs\") pod \"1a0b68e2-5f67-415a-96d0-8186134792bc\" (UID: \"1a0b68e2-5f67-415a-96d0-8186134792bc\") "
Nov 03 20:02:59.152402 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:59.152354    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a0b68e2-5f67-415a-96d0-8186134792bc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a0b68e2-5f67-415a-96d0-8186134792bc-maas-api-tls\") pod \"1a0b68e2-5f67-415a-96d0-8186134792bc\" (UID: \"1a0b68e2-5f67-415a-96d0-8186134792bc\") "
Nov 03 20:02:59.152402 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:59.152396    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a0b68e2-5f67-415a-96d0-8186134792bc-kube-api-access-lhh8b\" (UniqueName: \"kubernetes.io/projected/1a0b68e2-5f67-415a-96d0-8186134792bc-kube-api-access-lhh8b\") pod \"1a0b68e2-5f67-415a-96d0-8186134792bc\" (UID: \"1a0b68e2-5f67-415a-96d0-8186134792bc\") "
Nov 03 20:02:59.152600 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:59.152575    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a0b68e2-5f67-415a-96d0-8186134792bc-openshift-service-ca" pod "1a0b68e2-5f67-415a-96d0-8186134792bc" (UID: "1a0b68e2-5f67-415a-96d0-8186134792bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:02:59.153115 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:59.153090    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a0b68e2-5f67-415a-96d0-8186134792bc-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:59.154599 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:59.154573    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a0b68e2-5f67-415a-96d0-8186134792bc-kube-api-access-lhh8b" pod "1a0b68e2-5f67-415a-96d0-8186134792bc" (UID: "1a0b68e2-5f67-415a-96d0-8186134792bc"). InnerVolumeSpecName "kube-api-access-lhh8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:02:59.154694 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:59.154618    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a0b68e2-5f67-415a-96d0-8186134792bc-maas-api-tls" pod "1a0b68e2-5f67-415a-96d0-8186134792bc" (UID: "1a0b68e2-5f67-415a-96d0-8186134792bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:02:59.154748 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:59.154717    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a0b68e2-5f67-415a-96d0-8186134792bc-metrics-certs" pod "1a0b68e2-5f67-415a-96d0-8186134792bc" (UID: "1a0b68e2-5f67-415a-96d0-8186134792bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:02:59.259233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:59.256146    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a0b68e2-5f67-415a-96d0-8186134792bc-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:59.259233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:59.256183    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a0b68e2-5f67-415a-96d0-8186134792bc-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:59.259233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:59.256217    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhh8b\" (UniqueName: \"kubernetes.io/projected/1a0b68e2-5f67-415a-96d0-8186134792bc-kube-api-access-lhh8b\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:02:59.350814 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:02:59.350770    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c9a677a-d699-4b95-b217-09c48c89dff5" path="/var/lib/kubelet/pods/3c9a677a-d699-4b95-b217-09c48c89dff5/volumes"
Nov 03 20:03:00.042854 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:00.042820    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c"
Nov 03 20:03:00.046890 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:00.044553    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c" event={"ID":"1a0b68e2-5f67-415a-96d0-8186134792bc","Type":"ContainerDied","Data":"9b1eb91707a435bad587305a621207edde5099120609e2fa0f66e2af3193ecf1"}
Nov 03 20:03:00.048784 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:00.048728    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:03:00.048784 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:00.048782    5591 scope.go:122] "RemoveContainer" containerID="8b987c3f3c8ee227f46723ca2c8809b1b8f59b99eeb21393b39233e165192723"
Nov 03 20:03:00.048995 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:00.048904    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:03:00.093375 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:00.092802    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c"]
Nov 03 20:03:00.101111 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:00.101077    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-8fa615-5b94b6dbdd-87j2c"]
Nov 03 20:03:01.351777 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:01.351747    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a0b68e2-5f67-415a-96d0-8186134792bc" path="/var/lib/kubelet/pods/1a0b68e2-5f67-415a-96d0-8186134792bc/volumes"
Nov 03 20:03:02.890030 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:02.890002    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg"]
Nov 03 20:03:02.890936 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:02.890916    5591 cpu_manager.go:414] "containerMap: removing container" podUID="3c9a677a-d699-4b95-b217-09c48c89dff5" containerName="istio-proxy"
Nov 03 20:03:02.890936 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:02.890936    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c9a677a-d699-4b95-b217-09c48c89dff5" containerName="istio-proxy"
Nov 03 20:03:02.891111 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:02.890946    5591 cpu_manager.go:414] "containerMap: removing container" podUID="1a0b68e2-5f67-415a-96d0-8186134792bc" containerName="maas-api"
Nov 03 20:03:02.891111 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:02.890954    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a0b68e2-5f67-415a-96d0-8186134792bc" containerName="maas-api"
Nov 03 20:03:02.891268 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:02.891113    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c9a677a-d699-4b95-b217-09c48c89dff5" containerName="istio-proxy"
Nov 03 20:03:02.891268 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:02.891131    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a0b68e2-5f67-415a-96d0-8186134792bc" containerName="maas-api"
Nov 03 20:03:02.896721 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:02.896579    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg"
Nov 03 20:03:02.903142 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:02.902785    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-4fb1d0\""
Nov 03 20:03:02.903142 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:02.902879    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-4fb1d0-dockercfg-gdcgk\""
Nov 03 20:03:02.906703 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:02.905792    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz"]
Nov 03 20:03:02.912631 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:02.912497    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg"]
Nov 03 20:03:02.912631 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:02.912611    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz"
Nov 03 20:03:02.926090 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:02.921043    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2339afa1-517b-49cc-9d17-a037371e38e8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg\" (UID: \"2339afa1-517b-49cc-9d17-a037371e38e8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg"
Nov 03 20:03:02.926090 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:02.921124    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwtkc\" (UniqueName: \"kubernetes.io/projected/2339afa1-517b-49cc-9d17-a037371e38e8-kube-api-access-cwtkc\") pod \"payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg\" (UID: \"2339afa1-517b-49cc-9d17-a037371e38e8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg"
Nov 03 20:03:02.926090 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:02.925723    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz"]
Nov 03 20:03:03.059228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.058366    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2339afa1-517b-49cc-9d17-a037371e38e8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg\" (UID: \"2339afa1-517b-49cc-9d17-a037371e38e8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg"
Nov 03 20:03:03.059228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.058448    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75752636-c0d6-46d6-b2d2-da4122853fac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz\" (UID: \"75752636-c0d6-46d6-b2d2-da4122853fac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz"
Nov 03 20:03:03.059228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.058504    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwtkc\" (UniqueName: \"kubernetes.io/projected/2339afa1-517b-49cc-9d17-a037371e38e8-kube-api-access-cwtkc\") pod \"payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg\" (UID: \"2339afa1-517b-49cc-9d17-a037371e38e8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg"
Nov 03 20:03:03.059228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.058633    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpkmn\" (UniqueName: \"kubernetes.io/projected/75752636-c0d6-46d6-b2d2-da4122853fac-kube-api-access-fpkmn\") pod \"payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz\" (UID: \"75752636-c0d6-46d6-b2d2-da4122853fac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz"
Nov 03 20:03:03.134814 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.127983    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2339afa1-517b-49cc-9d17-a037371e38e8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg\" (UID: \"2339afa1-517b-49cc-9d17-a037371e38e8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg"
Nov 03 20:03:03.195945 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.193096    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwtkc\" (UniqueName: \"kubernetes.io/projected/2339afa1-517b-49cc-9d17-a037371e38e8-kube-api-access-cwtkc\") pod \"payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg\" (UID: \"2339afa1-517b-49cc-9d17-a037371e38e8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg"
Nov 03 20:03:03.195945 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.194564    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpkmn\" (UniqueName: \"kubernetes.io/projected/75752636-c0d6-46d6-b2d2-da4122853fac-kube-api-access-fpkmn\") pod \"payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz\" (UID: \"75752636-c0d6-46d6-b2d2-da4122853fac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz"
Nov 03 20:03:03.195945 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.194759    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75752636-c0d6-46d6-b2d2-da4122853fac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz\" (UID: \"75752636-c0d6-46d6-b2d2-da4122853fac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz"
Nov 03 20:03:03.195945 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.195581    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75752636-c0d6-46d6-b2d2-da4122853fac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz\" (UID: \"75752636-c0d6-46d6-b2d2-da4122853fac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz"
Nov 03 20:03:03.222882 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.222847    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg"
Nov 03 20:03:03.225282 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.225247    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpkmn\" (UniqueName: \"kubernetes.io/projected/75752636-c0d6-46d6-b2d2-da4122853fac-kube-api-access-fpkmn\") pod \"payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz\" (UID: \"75752636-c0d6-46d6-b2d2-da4122853fac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz"
Nov 03 20:03:03.283539 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.283508    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz"
Nov 03 20:03:03.296368 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.296332    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws"]
Nov 03 20:03:03.315613 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.303284    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws"
Nov 03 20:03:03.332333 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.328519    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-942930-serving-cert\""
Nov 03 20:03:03.332333 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.329303    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws"]
Nov 03 20:03:03.499760 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.499678    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws\" (UID: \"1b6e6abf-d08a-4f38-8ad7-6f24074355dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws"
Nov 03 20:03:03.499760 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.499720    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w6lm\" (UniqueName: \"kubernetes.io/projected/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-kube-api-access-5w6lm\") pod \"maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws\" (UID: \"1b6e6abf-d08a-4f38-8ad7-6f24074355dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws"
Nov 03 20:03:03.499760 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.499741    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws\" (UID: \"1b6e6abf-d08a-4f38-8ad7-6f24074355dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws"
Nov 03 20:03:03.499977 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.499943    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws\" (UID: \"1b6e6abf-d08a-4f38-8ad7-6f24074355dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws"
Nov 03 20:03:03.606227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.601302    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws\" (UID: \"1b6e6abf-d08a-4f38-8ad7-6f24074355dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws"
Nov 03 20:03:03.606227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.601507    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws\" (UID: \"1b6e6abf-d08a-4f38-8ad7-6f24074355dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws"
Nov 03 20:03:03.606227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.601545    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w6lm\" (UniqueName: \"kubernetes.io/projected/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-kube-api-access-5w6lm\") pod \"maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws\" (UID: \"1b6e6abf-d08a-4f38-8ad7-6f24074355dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws"
Nov 03 20:03:03.606227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.601575    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws\" (UID: \"1b6e6abf-d08a-4f38-8ad7-6f24074355dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws"
Nov 03 20:03:03.606227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.602724    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws\" (UID: \"1b6e6abf-d08a-4f38-8ad7-6f24074355dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws"
Nov 03 20:03:03.625229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.624953    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws\" (UID: \"1b6e6abf-d08a-4f38-8ad7-6f24074355dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws"
Nov 03 20:03:03.640427 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.639069    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg"]
Nov 03 20:03:03.655685 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.653806    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws\" (UID: \"1b6e6abf-d08a-4f38-8ad7-6f24074355dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws"
Nov 03 20:03:03.666472 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.666443    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz"]
Nov 03 20:03:03.669659 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.669592    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w6lm\" (UniqueName: \"kubernetes.io/projected/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-kube-api-access-5w6lm\") pod \"maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws\" (UID: \"1b6e6abf-d08a-4f38-8ad7-6f24074355dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws"
Nov 03 20:03:03.670183 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:03:03.670160    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75752636_c0d6_46d6_b2d2_da4122853fac.slice/crio-8360583ae05bfdcabee0b6307075ddc88c3a976a93ad6fa0922788166bb79a8f WatchSource:0}: Error finding container 8360583ae05bfdcabee0b6307075ddc88c3a976a93ad6fa0922788166bb79a8f: Status 404 returned error can't find the container with id 8360583ae05bfdcabee0b6307075ddc88c3a976a93ad6fa0922788166bb79a8f
Nov 03 20:03:03.671423 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:03:03.671396    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2339afa1_517b_49cc_9d17_a037371e38e8.slice/crio-0895916030a8f3980de5afd5724ab5cb71044e2f3706852065cc184fee8d0e0c WatchSource:0}: Error finding container 0895916030a8f3980de5afd5724ab5cb71044e2f3706852065cc184fee8d0e0c: Status 404 returned error can't find the container with id 0895916030a8f3980de5afd5724ab5cb71044e2f3706852065cc184fee8d0e0c
Nov 03 20:03:03.681334 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.681309    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws"
Nov 03 20:03:03.795441 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.792426    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"]
Nov 03 20:03:03.795441 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:03.792672    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght" podUID="b12a508d-cf94-477f-8a7b-856231ac6aa5" containerName="istio-proxy" containerID="cri-o://b2c874104a14463063f957077b665b15cd9dc02123533f91fed166aa36db1506" gracePeriod=30
Nov 03 20:03:04.000639 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:04.000608    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws"]
Nov 03 20:03:04.001313 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:03:04.001278    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6e6abf_d08a_4f38_8ad7_6f24074355dd.slice/crio-c4bdbf1a2b822cc3232320d1448b69212a718b905ad4b120147ebf64ee104679 WatchSource:0}: Error finding container c4bdbf1a2b822cc3232320d1448b69212a718b905ad4b120147ebf64ee104679: Status 404 returned error can't find the container with id c4bdbf1a2b822cc3232320d1448b69212a718b905ad4b120147ebf64ee104679
Nov 03 20:03:04.274292 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:04.274175    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz" event={"ID":"75752636-c0d6-46d6-b2d2-da4122853fac","Type":"ContainerStarted","Data":"d13147fa6cf8257ec8edeaea1229c525de295de0cda7bd5f71f508f996687fd3"}
Nov 03 20:03:04.274574 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:04.274553    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz" event={"ID":"75752636-c0d6-46d6-b2d2-da4122853fac","Type":"ContainerStarted","Data":"8360583ae05bfdcabee0b6307075ddc88c3a976a93ad6fa0922788166bb79a8f"}
Nov 03 20:03:04.275731 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:04.275709    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz"
Nov 03 20:03:04.277928 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:04.277907    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg" event={"ID":"2339afa1-517b-49cc-9d17-a037371e38e8","Type":"ContainerStarted","Data":"b09ca31bbf40bcec53b3632b26ba4f993271685fe8afd7162539a960a41ba789"}
Nov 03 20:03:04.278107 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:04.278093    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg" event={"ID":"2339afa1-517b-49cc-9d17-a037371e38e8","Type":"ContainerStarted","Data":"0895916030a8f3980de5afd5724ab5cb71044e2f3706852065cc184fee8d0e0c"}
Nov 03 20:03:04.279118 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:04.279096    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg"
Nov 03 20:03:04.306196 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:04.301352    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws" event={"ID":"1b6e6abf-d08a-4f38-8ad7-6f24074355dd","Type":"ContainerStarted","Data":"c4bdbf1a2b822cc3232320d1448b69212a718b905ad4b120147ebf64ee104679"}
Nov 03 20:03:04.306587 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:04.306530    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz" podStartSLOduration=2.306509351 podStartE2EDuration="2.306509351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:04.296849639 +0000 UTC m=+1533.069142658" watchObservedRunningTime="2026-11-03 20:03:04.306509351 +0000 UTC m=+1533.078802369"
Nov 03 20:03:04.318694 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:04.318631    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg" podStartSLOduration=2.318616126 podStartE2EDuration="2.318616126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:04.317625913 +0000 UTC m=+1533.089918931" watchObservedRunningTime="2026-11-03 20:03:04.318616126 +0000 UTC m=+1533.090909138"
Nov 03 20:03:05.309848 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.309808    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws" event={"ID":"1b6e6abf-d08a-4f38-8ad7-6f24074355dd","Type":"ContainerStarted","Data":"6688653124a3e4a7eac13bf951c1d39ac6557ee549a90c069bb1a3ec1bded20d"}
Nov 03 20:03:05.310373 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.310097    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws"
Nov 03 20:03:05.334126 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.334072    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws" podStartSLOduration=1.765288379 podStartE2EDuration="2.33405726s" totalImagesPullingTime="568.768881ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:05.330523559 +0000 UTC m=+1534.102816576" watchObservedRunningTime="2026-11-03 20:03:05.33405726 +0000 UTC m=+1534.106350634"
Nov 03 20:03:05.383939 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.383903    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"]
Nov 03 20:03:05.388877 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.388851    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.429308 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.419306    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"]
Nov 03 20:03:05.443052 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.443007    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.443052 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.443043    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpccm\" (UniqueName: \"kubernetes.io/projected/2b1c87e7-ecf0-438f-a3c4-93da53813516-kube-api-access-dpccm\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.443347 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.443070    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.443347 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.443161    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1c87e7-ecf0-438f-a3c4-93da53813516-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.443347 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.443227    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.443347 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.443270    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.443715 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.443380    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.443715 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.443544    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.443830 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.443667    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.548221 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.548178    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.548522 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.548309    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.548522 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.548405    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.548522 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.548434    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpccm\" (UniqueName: \"kubernetes.io/projected/2b1c87e7-ecf0-438f-a3c4-93da53813516-kube-api-access-dpccm\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.548522 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.548475    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.548522 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.548513    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1c87e7-ecf0-438f-a3c4-93da53813516-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.548822 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.548544    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.548822 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.548585    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.548822 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.548653    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.548822 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.548734    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-data\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.549027 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.548968    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-credential-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.549316 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.549287    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-workload-certs\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.549554 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.549534    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-workload-socket\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.549697 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.549676    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1c87e7-ecf0-438f-a3c4-93da53813516-istiod-ca-cert\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.552349 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.552323    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-envoy\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.552349 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.552345    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-podinfo\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.570672 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.570612    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpccm\" (UniqueName: \"kubernetes.io/projected/2b1c87e7-ecf0-438f-a3c4-93da53813516-kube-api-access-dpccm\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.571651 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.571628    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-token\") pod \"e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") " pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:05.716153 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:05.716119    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:06.011446 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:03:06.011419    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1c87e7_ecf0_438f_a3c4_93da53813516.slice/crio-b9b63d65772a2285af50eab871be450527504c0df71b3e5eac5fcc132b44b1d5 WatchSource:0}: Error finding container b9b63d65772a2285af50eab871be450527504c0df71b3e5eac5fcc132b44b1d5: Status 404 returned error can't find the container with id b9b63d65772a2285af50eab871be450527504c0df71b3e5eac5fcc132b44b1d5
Nov 03 20:03:06.012754 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:06.012731    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"]
Nov 03 20:03:06.014063 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:06.014032    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:03:06.014145 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:06.014095    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:03:06.014145 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:06.014126    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:03:06.344513 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:06.344412    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n" event={"ID":"2b1c87e7-ecf0-438f-a3c4-93da53813516","Type":"ContainerStarted","Data":"968872f53ed39a1ad4244ad035af2045618c1bf05eedd7b4b81b430ffb55b8cf"}
Nov 03 20:03:06.345130 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:06.345083    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n" event={"ID":"2b1c87e7-ecf0-438f-a3c4-93da53813516","Type":"ContainerStarted","Data":"b9b63d65772a2285af50eab871be450527504c0df71b3e5eac5fcc132b44b1d5"}
Nov 03 20:03:06.371485 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:06.371435    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n" podStartSLOduration=1.371421624 podStartE2EDuration="1.371421624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:06.367785105 +0000 UTC m=+1535.140078126" watchObservedRunningTime="2026-11-03 20:03:06.371421624 +0000 UTC m=+1535.143714636"
Nov 03 20:03:06.717458 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:06.717305    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:07.751229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:07.744381    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 03 20:03:07.751229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:07.744467    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n" podUID="2b1c87e7-ecf0-438f-a3c4-93da53813516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 03 20:03:08.459270 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:08.438387    5591 patch_prober.go:29] interesting pod/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Nov 03 20:03:08.459270 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:08.438470    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght" podUID="b12a508d-cf94-477f-8a7b-856231ac6aa5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:03:08.716803 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:08.716692    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 03 20:03:08.717076 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:08.717039    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n" podUID="2b1c87e7-ecf0-438f-a3c4-93da53813516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 03 20:03:09.119232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:09.118386    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:03:09.171232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:09.167362    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:09.171232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:09.168508    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded"
Nov 03 20:03:09.447030 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:09.445965    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:03:09.462157 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:09.461860    5591 generic.go:356] "Generic (PLEG): container finished" podID="b12a508d-cf94-477f-8a7b-856231ac6aa5" containerID="b2c874104a14463063f957077b665b15cd9dc02123533f91fed166aa36db1506" exitCode=0
Nov 03 20:03:09.462157 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:09.461954    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght" event={"ID":"b12a508d-cf94-477f-8a7b-856231ac6aa5","Type":"ContainerDied","Data":"b2c874104a14463063f957077b665b15cd9dc02123533f91fed166aa36db1506"}
Nov 03 20:03:09.462157 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:09.462017    5591 scope.go:122] "RemoveContainer" containerID="431e5b0f265a02c8259b3be9f7f15edfc112cb2c0f3d46655f0b2cfd561be69a"
Nov 03 20:03:09.809231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:09.808876    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 20:03:10.041331 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:09.811091    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:10.041331 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:09.811566    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 20:03:10.041331 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:09.827133    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 03 20:03:10.041331 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:09.827190    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n" podUID="2b1c87e7-ecf0-438f-a3c4-93da53813516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 03 20:03:10.041331 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:09.942388    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:03:10.174302 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.167314    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:10.174302 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.168377    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:10.205230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.182127    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 20:03:10.205230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.182198    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 20:03:10.426889 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.425152    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:03:10.458251 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.453483    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:10.509270 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.459189    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 20:03:10.755490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.755392    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 03 20:03:10.769516 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.769467    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n" podUID="2b1c87e7-ecf0-438f-a3c4-93da53813516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 03 20:03:10.799702 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.798404    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:10.870230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.869509    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b12a508d-cf94-477f-8a7b-856231ac6aa5-kube-api-access-2mjmk\" (UniqueName: \"kubernetes.io/projected/b12a508d-cf94-477f-8a7b-856231ac6aa5-kube-api-access-2mjmk\") pod \"b12a508d-cf94-477f-8a7b-856231ac6aa5\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") "
Nov 03 20:03:10.870230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.869569    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-data\") pod \"b12a508d-cf94-477f-8a7b-856231ac6aa5\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") "
Nov 03 20:03:10.870230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.869630    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-podinfo\") pod \"b12a508d-cf94-477f-8a7b-856231ac6aa5\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") "
Nov 03 20:03:10.870230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.869679    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-token\" (UniqueName: \"kubernetes.io/projected/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-token\") pod \"b12a508d-cf94-477f-8a7b-856231ac6aa5\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") "
Nov 03 20:03:10.870230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.869724    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-envoy\") pod \"b12a508d-cf94-477f-8a7b-856231ac6aa5\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") "
Nov 03 20:03:10.870230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.869790    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b12a508d-cf94-477f-8a7b-856231ac6aa5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12a508d-cf94-477f-8a7b-856231ac6aa5-istiod-ca-cert\") pod \"b12a508d-cf94-477f-8a7b-856231ac6aa5\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") "
Nov 03 20:03:10.870230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.869848    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-workload-socket\") pod \"b12a508d-cf94-477f-8a7b-856231ac6aa5\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") "
Nov 03 20:03:10.870230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.869899    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-workload-certs\") pod \"b12a508d-cf94-477f-8a7b-856231ac6aa5\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") "
Nov 03 20:03:10.870230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.869928    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-credential-socket\") pod \"b12a508d-cf94-477f-8a7b-856231ac6aa5\" (UID: \"b12a508d-cf94-477f-8a7b-856231ac6aa5\") "
Nov 03 20:03:10.980229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.894311    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:10.980229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.897522    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b12a508d-cf94-477f-8a7b-856231ac6aa5-istiod-ca-cert" pod "b12a508d-cf94-477f-8a7b-856231ac6aa5" (UID: "b12a508d-cf94-477f-8a7b-856231ac6aa5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:03:10.980229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.897886    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-data" pod "b12a508d-cf94-477f-8a7b-856231ac6aa5" (UID: "b12a508d-cf94-477f-8a7b-856231ac6aa5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:10.980229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.898433    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 20:03:10.980229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.900530    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 20:03:10.980229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.900596    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 20:03:10.980229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.951368    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:03:10.980229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.951431    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb" podUID="fa169e55-a457-4166-88ab-c65b6081d08e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:10.985779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:10.985571    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-workload-socket" pod "b12a508d-cf94-477f-8a7b-856231ac6aa5" (UID: "b12a508d-cf94-477f-8a7b-856231ac6aa5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:11.026292 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:11.008313    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-credential-socket" pod "b12a508d-cf94-477f-8a7b-856231ac6aa5" (UID: "b12a508d-cf94-477f-8a7b-856231ac6aa5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:11.026292 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:11.009367    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12a508d-cf94-477f-8a7b-856231ac6aa5-kube-api-access-2mjmk" pod "b12a508d-cf94-477f-8a7b-856231ac6aa5" (UID: "b12a508d-cf94-477f-8a7b-856231ac6aa5"). InnerVolumeSpecName "kube-api-access-2mjmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:11.026292 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:11.009461    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-token" pod "b12a508d-cf94-477f-8a7b-856231ac6aa5" (UID: "b12a508d-cf94-477f-8a7b-856231ac6aa5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:11.057312 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:11.029229    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:03:11.058289 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:11.058258    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 20:03:11.152265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:11.117327    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mjmk\" (UniqueName: \"kubernetes.io/projected/b12a508d-cf94-477f-8a7b-856231ac6aa5-kube-api-access-2mjmk\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:11.152265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:11.117363    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:11.152265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:11.117381    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:11.152265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:11.117396    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12a508d-cf94-477f-8a7b-856231ac6aa5-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:11.152265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:11.117409    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:11.152265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:11.117423    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:11.190683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:11.183930    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:11.287811 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:11.287699    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf"
Nov 03 20:03:11.289337 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:11.289307    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:03:11.443290 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:11.352872    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght" event={"ID":"b12a508d-cf94-477f-8a7b-856231ac6aa5","Type":"ContainerDied","Data":"9d0ad6bd0d7bf1c8a44d32a16aa8a529bfd0cb0b625716f69ab07a3e41dfd379"}
Nov 03 20:03:11.443290 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:11.443012    5591 scope.go:122] "RemoveContainer" containerID="b2c874104a14463063f957077b665b15cd9dc02123533f91fed166aa36db1506"
Nov 03 20:03:11.443290 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:11.352971    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"
Nov 03 20:03:11.443290 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:11.420307    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-workload-certs" pod "b12a508d-cf94-477f-8a7b-856231ac6aa5" (UID: "b12a508d-cf94-477f-8a7b-856231ac6aa5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:11.680292 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:11.676819    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:11.845448 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:11.845404    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:11.884432 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:11.884396    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-envoy" pod "b12a508d-cf94-477f-8a7b-856231ac6aa5" (UID: "b12a508d-cf94-477f-8a7b-856231ac6aa5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:03:11.920457 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:11.919660    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:11.920457 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:11.919725    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 20:03:11.920457 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:11.919966    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 20:03:11.967425 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:11.967354    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-podinfo" pod "b12a508d-cf94-477f-8a7b-856231ac6aa5" (UID: "b12a508d-cf94-477f-8a7b-856231ac6aa5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:03:11.978240 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:11.974752    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 20:03:12.052239 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:12.007313    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl"
Nov 03 20:03:12.052239 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:12.012443    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws"
Nov 03 20:03:12.052239 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:12.013788    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 20:03:12.052239 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:12.019319    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 03 20:03:12.052239 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:12.019370    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n" podUID="2b1c87e7-ecf0-438f-a3c4-93da53813516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 03 20:03:12.052239 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:12.047533    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:03:12.052239 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:12.047603    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r" podUID="8e00f1a1-8156-4fa7-8308-b48127f82aa6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:12.082351 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:12.054819    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12a508d-cf94-477f-8a7b-856231ac6aa5-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:12.082351 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:12.071181    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:12.270983 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:12.270932    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:12.345035 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:12.344990    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:03:12.347367 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:12.347308    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v" podUID="6d7456ba-061d-4c8a-974c-326c72027557" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:12.547314 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:12.536414    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:03:12.547314 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:12.541335    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:13.017281 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:13.013147    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:13.123367 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:13.017679    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-c2fbm" podUID="0de7df58-cc36-454f-8954-fb6cb562da6a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:13.124427 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:13.124386    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-fxm7k" podUID="6efbc03a-f9b7-463d-969e-7c420f0d59e1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:13.142533 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:13.057599    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 20:03:13.147235 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:13.057327    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 20:03:13.237304 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:13.237254    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:13.558739 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:13.289584    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:13.586386 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:13.586336    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:03:13.702031 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:13.701985    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:03:13.893446 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:13.893341    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 03 20:03:13.893446 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:13.893421    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n" podUID="2b1c87e7-ecf0-438f-a3c4-93da53813516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 03 20:03:14.017234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.007591    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:14.039884 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:03:14.039850    5591 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.026s"
Nov 03 20:03:14.040126 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.040108    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h"]
Nov 03 20:03:14.144501 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.144354    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l"]
Nov 03 20:03:14.144501 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.144427    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8"]
Nov 03 20:03:14.145516 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.145489    5591 cpu_manager.go:414] "containerMap: removing container" podUID="b12a508d-cf94-477f-8a7b-856231ac6aa5" containerName="istio-proxy"
Nov 03 20:03:14.145516 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.145514    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b12a508d-cf94-477f-8a7b-856231ac6aa5" containerName="istio-proxy"
Nov 03 20:03:14.145704 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.145564    5591 cpu_manager.go:414] "containerMap: removing container" podUID="b12a508d-cf94-477f-8a7b-856231ac6aa5" containerName="istio-proxy"
Nov 03 20:03:14.145704 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.145574    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b12a508d-cf94-477f-8a7b-856231ac6aa5" containerName="istio-proxy"
Nov 03 20:03:14.145831 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.145768    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b12a508d-cf94-477f-8a7b-856231ac6aa5" containerName="istio-proxy"
Nov 03 20:03:14.145831 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.145788    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b12a508d-cf94-477f-8a7b-856231ac6aa5" containerName="istio-proxy"
Nov 03 20:03:14.151650 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.151628    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8"]
Nov 03 20:03:14.151880 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.151864    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 20:03:14.152010 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.151707    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8"
Nov 03 20:03:14.152559 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.152017    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 20:03:14.152703 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.152693    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 20:03:14.152794 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.152773    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 20:03:14.152858 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.152850    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"]
Nov 03 20:03:14.152933 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.152926    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 20:03:14.153628 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.153612    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 20:03:14.153750 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.153741    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 20:03:14.153835 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.153828    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 20:03:14.153899 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.153892    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-8fa615-openshift-default-5b9bd9b8f7-mfght"]
Nov 03 20:03:14.157055 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.156996    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-4fb1d0-serving-cert\""
Nov 03 20:03:14.229654 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.229586    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0366c67a-080b-4983-8057-b0d9ef562477-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8\" (UID: \"0366c67a-080b-4983-8057-b0d9ef562477\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8"
Nov 03 20:03:14.230092 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.230058    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4hdr\" (UniqueName: \"kubernetes.io/projected/0366c67a-080b-4983-8057-b0d9ef562477-kube-api-access-h4hdr\") pod \"maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8\" (UID: \"0366c67a-080b-4983-8057-b0d9ef562477\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8"
Nov 03 20:03:14.231532 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.231500    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0366c67a-080b-4983-8057-b0d9ef562477-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8\" (UID: \"0366c67a-080b-4983-8057-b0d9ef562477\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8"
Nov 03 20:03:14.231688 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.231660    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0366c67a-080b-4983-8057-b0d9ef562477-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8\" (UID: \"0366c67a-080b-4983-8057-b0d9ef562477\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8"
Nov 03 20:03:14.334738 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.334688    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4hdr\" (UniqueName: \"kubernetes.io/projected/0366c67a-080b-4983-8057-b0d9ef562477-kube-api-access-h4hdr\") pod \"maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8\" (UID: \"0366c67a-080b-4983-8057-b0d9ef562477\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8"
Nov 03 20:03:14.335318 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.335289    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0366c67a-080b-4983-8057-b0d9ef562477-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8\" (UID: \"0366c67a-080b-4983-8057-b0d9ef562477\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8"
Nov 03 20:03:14.335538 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.335521    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0366c67a-080b-4983-8057-b0d9ef562477-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8\" (UID: \"0366c67a-080b-4983-8057-b0d9ef562477\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8"
Nov 03 20:03:14.335855 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.335834    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0366c67a-080b-4983-8057-b0d9ef562477-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8\" (UID: \"0366c67a-080b-4983-8057-b0d9ef562477\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8"
Nov 03 20:03:14.337362 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.337339    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0366c67a-080b-4983-8057-b0d9ef562477-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8\" (UID: \"0366c67a-080b-4983-8057-b0d9ef562477\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8"
Nov 03 20:03:14.343524 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.343262    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0366c67a-080b-4983-8057-b0d9ef562477-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8\" (UID: \"0366c67a-080b-4983-8057-b0d9ef562477\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8"
Nov 03 20:03:14.348884 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.348859    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0366c67a-080b-4983-8057-b0d9ef562477-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8\" (UID: \"0366c67a-080b-4983-8057-b0d9ef562477\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8"
Nov 03 20:03:14.353458 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.353436    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4hdr\" (UniqueName: \"kubernetes.io/projected/0366c67a-080b-4983-8057-b0d9ef562477-kube-api-access-h4hdr\") pod \"maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8\" (UID: \"0366c67a-080b-4983-8057-b0d9ef562477\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8"
Nov 03 20:03:14.474562 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.474466    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8"
Nov 03 20:03:14.744353 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.724564    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 03 20:03:14.744353 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:14.724630    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n" podUID="2b1c87e7-ecf0-438f-a3c4-93da53813516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 03 20:03:15.014590 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:15.014549    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8"]
Nov 03 20:03:15.127188 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:15.127153    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8" event={"ID":"0366c67a-080b-4983-8057-b0d9ef562477","Type":"ContainerStarted","Data":"c9e19c1a3528ce5ba9490f20319a457d2e575ccdf2a1c8be46821f656f6e392e"}
Nov 03 20:03:15.473082 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:15.472987    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b12a508d-cf94-477f-8a7b-856231ac6aa5" path="/var/lib/kubelet/pods/b12a508d-cf94-477f-8a7b-856231ac6aa5/volumes"
Nov 03 20:03:15.727570 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:15.727473    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 03 20:03:15.727877 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:15.727839    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n" podUID="2b1c87e7-ecf0-438f-a3c4-93da53813516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 03 20:03:16.139293 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:16.139256    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8" event={"ID":"0366c67a-080b-4983-8057-b0d9ef562477","Type":"ContainerStarted","Data":"006f25552fde813c3298da814e906a6144795fbd26cf455a96c17dc244d0797f"}
Nov 03 20:03:16.139702 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:16.139316    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8"
Nov 03 20:03:16.166624 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:16.166574    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8" podStartSLOduration=4.610654914 podStartE2EDuration="5.166558745s" totalImagesPullingTime="555.903831ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:16.163916808 +0000 UTC m=+1544.936209828" watchObservedRunningTime="2026-11-03 20:03:16.166558745 +0000 UTC m=+1544.938851763"
Nov 03 20:03:16.366230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:16.363881    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz"
Nov 03 20:03:16.378932 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:16.378589    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg"
Nov 03 20:03:16.717226 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:16.716687    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body=
Nov 03 20:03:16.717226 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:16.716760    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n" podUID="2b1c87e7-ecf0-438f-a3c4-93da53813516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused"
Nov 03 20:03:18.757402 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:18.757361    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:03:18.757809 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:18.757449    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n" podUID="2b1c87e7-ecf0-438f-a3c4-93da53813516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:19.102376 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:19.102317    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h" podUID="221d9a79-e987-4b03-bd4f-e1f6718fe2ce" containerName="payload-processing" containerID="cri-o://7100a93dcba89dc90935f55ce5f558688fa1c946c18f95e6fb6241d170cc9998" gracePeriod=25
Nov 03 20:03:19.160283 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:19.151283    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l" podUID="f72239c9-93c7-4f75-8296-3c8816841ac6" containerName="payload-pre-processing" containerID="cri-o://9228be36e0c2be328bb3f682348d2952b36d6ee8b7364822857923939e2c9831" gracePeriod=25
Nov 03 20:03:19.412648 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:19.411678    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:19.416444 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:19.413758    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:19.416444 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:19.415307    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:03:19.418917 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:19.418592    5591 generic.go:356] "Generic (PLEG): container finished" podID="f72239c9-93c7-4f75-8296-3c8816841ac6" containerID="9228be36e0c2be328bb3f682348d2952b36d6ee8b7364822857923939e2c9831" exitCode=0
Nov 03 20:03:19.418917 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:19.418777    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l" event={"ID":"f72239c9-93c7-4f75-8296-3c8816841ac6","Type":"ContainerDied","Data":"9228be36e0c2be328bb3f682348d2952b36d6ee8b7364822857923939e2c9831"}
Nov 03 20:03:19.421606 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:19.421577    5591 generic.go:356] "Generic (PLEG): container finished" podID="221d9a79-e987-4b03-bd4f-e1f6718fe2ce" containerID="7100a93dcba89dc90935f55ce5f558688fa1c946c18f95e6fb6241d170cc9998" exitCode=0
Nov 03 20:03:19.421733 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:19.421664    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h" event={"ID":"221d9a79-e987-4b03-bd4f-e1f6718fe2ce","Type":"ContainerDied","Data":"7100a93dcba89dc90935f55ce5f558688fa1c946c18f95e6fb6241d170cc9998"}
Nov 03 20:03:19.499117 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:19.499087    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h"
Nov 03 20:03:19.516121 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:19.516085    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/221d9a79-e987-4b03-bd4f-e1f6718fe2ce-kube-api-access-fp2r6\" (UniqueName: \"kubernetes.io/projected/221d9a79-e987-4b03-bd4f-e1f6718fe2ce-kube-api-access-fp2r6\") pod \"221d9a79-e987-4b03-bd4f-e1f6718fe2ce\" (UID: \"221d9a79-e987-4b03-bd4f-e1f6718fe2ce\") "
Nov 03 20:03:19.516329 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:19.516310    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/221d9a79-e987-4b03-bd4f-e1f6718fe2ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221d9a79-e987-4b03-bd4f-e1f6718fe2ce-plugins-config-volume\") pod \"221d9a79-e987-4b03-bd4f-e1f6718fe2ce\" (UID: \"221d9a79-e987-4b03-bd4f-e1f6718fe2ce\") "
Nov 03 20:03:19.516683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:19.516635    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/221d9a79-e987-4b03-bd4f-e1f6718fe2ce-plugins-config-volume" pod "221d9a79-e987-4b03-bd4f-e1f6718fe2ce" (UID: "221d9a79-e987-4b03-bd4f-e1f6718fe2ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:03:19.517263 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:19.517238    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/221d9a79-e987-4b03-bd4f-e1f6718fe2ce-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:19.518404 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:19.518380    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/221d9a79-e987-4b03-bd4f-e1f6718fe2ce-kube-api-access-fp2r6" pod "221d9a79-e987-4b03-bd4f-e1f6718fe2ce" (UID: "221d9a79-e987-4b03-bd4f-e1f6718fe2ce"). InnerVolumeSpecName "kube-api-access-fp2r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:19.557097 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:19.556869    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l"
Nov 03 20:03:19.617997 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:19.617961    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f72239c9-93c7-4f75-8296-3c8816841ac6-kube-api-access-hjmz8\" (UniqueName: \"kubernetes.io/projected/f72239c9-93c7-4f75-8296-3c8816841ac6-kube-api-access-hjmz8\") pod \"f72239c9-93c7-4f75-8296-3c8816841ac6\" (UID: \"f72239c9-93c7-4f75-8296-3c8816841ac6\") "
Nov 03 20:03:19.619039 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:19.618132    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f72239c9-93c7-4f75-8296-3c8816841ac6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f72239c9-93c7-4f75-8296-3c8816841ac6-plugins-config-volume\") pod \"f72239c9-93c7-4f75-8296-3c8816841ac6\" (UID: \"f72239c9-93c7-4f75-8296-3c8816841ac6\") "
Nov 03 20:03:19.619039 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:19.618910    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp2r6\" (UniqueName: \"kubernetes.io/projected/221d9a79-e987-4b03-bd4f-e1f6718fe2ce-kube-api-access-fp2r6\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:19.619246 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:19.619221    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72239c9-93c7-4f75-8296-3c8816841ac6-plugins-config-volume" pod "f72239c9-93c7-4f75-8296-3c8816841ac6" (UID: "f72239c9-93c7-4f75-8296-3c8816841ac6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:03:19.621524 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:19.621457    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72239c9-93c7-4f75-8296-3c8816841ac6-kube-api-access-hjmz8" pod "f72239c9-93c7-4f75-8296-3c8816841ac6" (UID: "f72239c9-93c7-4f75-8296-3c8816841ac6"). InnerVolumeSpecName "kube-api-access-hjmz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:19.752362 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:19.752157    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjmz8\" (UniqueName: \"kubernetes.io/projected/f72239c9-93c7-4f75-8296-3c8816841ac6-kube-api-access-hjmz8\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:19.752362 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:19.752217    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f72239c9-93c7-4f75-8296-3c8816841ac6-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:20.432216 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.432183    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h"
Nov 03 20:03:20.432626 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.432234    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h" event={"ID":"221d9a79-e987-4b03-bd4f-e1f6718fe2ce","Type":"ContainerDied","Data":"3ab2888d2a7574e5bb6a90b261a22efef4c7963046e067908141054538c1795c"}
Nov 03 20:03:20.432626 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.432298    5591 scope.go:122] "RemoveContainer" containerID="7100a93dcba89dc90935f55ce5f558688fa1c946c18f95e6fb6241d170cc9998"
Nov 03 20:03:20.433976 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.433953    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l" event={"ID":"f72239c9-93c7-4f75-8296-3c8816841ac6","Type":"ContainerDied","Data":"154cebee71a013cde1b5a66a9d7e17cb8fc97046f8581057e4e8fc36e4164b4c"}
Nov 03 20:03:20.434136 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.434118    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l"
Nov 03 20:03:20.444987 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.444855    5591 scope.go:122] "RemoveContainer" containerID="9228be36e0c2be328bb3f682348d2952b36d6ee8b7364822857923939e2c9831"
Nov 03 20:03:20.468802 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.468770    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h"]
Nov 03 20:03:20.481626 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.481599    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c131fa-576cccf4c6-7tr7h"]
Nov 03 20:03:20.492887 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.492849    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v"]
Nov 03 20:03:20.493627 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.493605    5591 cpu_manager.go:414] "containerMap: removing container" podUID="221d9a79-e987-4b03-bd4f-e1f6718fe2ce" containerName="payload-processing"
Nov 03 20:03:20.493627 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.493625    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="221d9a79-e987-4b03-bd4f-e1f6718fe2ce" containerName="payload-processing"
Nov 03 20:03:20.493792 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.493648    5591 cpu_manager.go:414] "containerMap: removing container" podUID="f72239c9-93c7-4f75-8296-3c8816841ac6" containerName="payload-pre-processing"
Nov 03 20:03:20.493792 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.493656    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f72239c9-93c7-4f75-8296-3c8816841ac6" containerName="payload-pre-processing"
Nov 03 20:03:20.493792 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.493785    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="221d9a79-e987-4b03-bd4f-e1f6718fe2ce" containerName="payload-processing"
Nov 03 20:03:20.493942 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.493795    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f72239c9-93c7-4f75-8296-3c8816841ac6" containerName="payload-pre-processing"
Nov 03 20:03:20.498861 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.498831    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v"
Nov 03 20:03:20.527387 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.527359    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l"]
Nov 03 20:03:20.530410 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.530390    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v"]
Nov 03 20:03:20.532497 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.532220    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-bfc240-dockercfg-9cmd2\""
Nov 03 20:03:20.535097 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.535076    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-bfc240\""
Nov 03 20:03:20.541572 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.541548    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c131fa-59c6c6877f-7xk9l"]
Nov 03 20:03:20.564254 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.564213    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4mqv\" (UniqueName: \"kubernetes.io/projected/26612e45-5a14-497a-9e90-06f336b9d0d1-kube-api-access-p4mqv\") pod \"payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v\" (UID: \"26612e45-5a14-497a-9e90-06f336b9d0d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v"
Nov 03 20:03:20.564506 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.564485    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26612e45-5a14-497a-9e90-06f336b9d0d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v\" (UID: \"26612e45-5a14-497a-9e90-06f336b9d0d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v"
Nov 03 20:03:20.575403 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.575369    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp"]
Nov 03 20:03:20.582587 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.582562    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp"
Nov 03 20:03:20.595894 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.595854    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp"]
Nov 03 20:03:20.666195 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.666148    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26612e45-5a14-497a-9e90-06f336b9d0d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v\" (UID: \"26612e45-5a14-497a-9e90-06f336b9d0d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v"
Nov 03 20:03:20.666195 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.666232    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqj6c\" (UniqueName: \"kubernetes.io/projected/e3c18563-3261-4841-81d2-3db2ec91abd3-kube-api-access-fqj6c\") pod \"payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp\" (UID: \"e3c18563-3261-4841-81d2-3db2ec91abd3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp"
Nov 03 20:03:20.666537 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.666328    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c18563-3261-4841-81d2-3db2ec91abd3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp\" (UID: \"e3c18563-3261-4841-81d2-3db2ec91abd3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp"
Nov 03 20:03:20.666537 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.666374    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4mqv\" (UniqueName: \"kubernetes.io/projected/26612e45-5a14-497a-9e90-06f336b9d0d1-kube-api-access-p4mqv\") pod \"payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v\" (UID: \"26612e45-5a14-497a-9e90-06f336b9d0d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v"
Nov 03 20:03:20.696276 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.695172    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26612e45-5a14-497a-9e90-06f336b9d0d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v\" (UID: \"26612e45-5a14-497a-9e90-06f336b9d0d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v"
Nov 03 20:03:20.767465 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.765457    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4mqv\" (UniqueName: \"kubernetes.io/projected/26612e45-5a14-497a-9e90-06f336b9d0d1-kube-api-access-p4mqv\") pod \"payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v\" (UID: \"26612e45-5a14-497a-9e90-06f336b9d0d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v"
Nov 03 20:03:20.812232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.812023    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c18563-3261-4841-81d2-3db2ec91abd3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp\" (UID: \"e3c18563-3261-4841-81d2-3db2ec91abd3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp"
Nov 03 20:03:20.816231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.812731    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqj6c\" (UniqueName: \"kubernetes.io/projected/e3c18563-3261-4841-81d2-3db2ec91abd3-kube-api-access-fqj6c\") pod \"payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp\" (UID: \"e3c18563-3261-4841-81d2-3db2ec91abd3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp"
Nov 03 20:03:20.833230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.831871    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c18563-3261-4841-81d2-3db2ec91abd3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp\" (UID: \"e3c18563-3261-4841-81d2-3db2ec91abd3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp"
Nov 03 20:03:20.833230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.832164    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v"
Nov 03 20:03:20.896858 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.896815    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqj6c\" (UniqueName: \"kubernetes.io/projected/e3c18563-3261-4841-81d2-3db2ec91abd3-kube-api-access-fqj6c\") pod \"payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp\" (UID: \"e3c18563-3261-4841-81d2-3db2ec91abd3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp"
Nov 03 20:03:20.907448 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:20.902769    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp"
Nov 03 20:03:21.176225 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.176167    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt"]
Nov 03 20:03:21.186833 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.186802    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt"
Nov 03 20:03:21.197637 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.197356    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt"]
Nov 03 20:03:21.197637 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.197616    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-f154ac\""
Nov 03 20:03:21.212468 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.212128    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-f154ac-dockercfg-x9sxz\""
Nov 03 20:03:21.221186 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.221006    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b4867c-02bd-4c72-8a46-6db1867a40c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt\" (UID: \"c9b4867c-02bd-4c72-8a46-6db1867a40c3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt"
Nov 03 20:03:21.221460 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.221282    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p9rh\" (UniqueName: \"kubernetes.io/projected/c9b4867c-02bd-4c72-8a46-6db1867a40c3-kube-api-access-8p9rh\") pod \"payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt\" (UID: \"c9b4867c-02bd-4c72-8a46-6db1867a40c3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt"
Nov 03 20:03:21.221873 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.221805    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v"]
Nov 03 20:03:21.232998 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.232970    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z"]
Nov 03 20:03:21.238846 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.238782    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z"
Nov 03 20:03:21.261833 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.261149    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z"]
Nov 03 20:03:21.266106 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.265999    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp"]
Nov 03 20:03:21.270316 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:03:21.269800    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c18563_3261_4841_81d2_3db2ec91abd3.slice/crio-f1605e8af12a8b828cc2ce9a4bfaa5f80b705a0546921c679861f060d42076a3 WatchSource:0}: Error finding container f1605e8af12a8b828cc2ce9a4bfaa5f80b705a0546921c679861f060d42076a3: Status 404 returned error can't find the container with id f1605e8af12a8b828cc2ce9a4bfaa5f80b705a0546921c679861f060d42076a3
Nov 03 20:03:21.322439 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.322409    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p9rh\" (UniqueName: \"kubernetes.io/projected/c9b4867c-02bd-4c72-8a46-6db1867a40c3-kube-api-access-8p9rh\") pod \"payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt\" (UID: \"c9b4867c-02bd-4c72-8a46-6db1867a40c3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt"
Nov 03 20:03:21.322601 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.322496    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z\" (UID: \"7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z"
Nov 03 20:03:21.322720 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.322696    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b4867c-02bd-4c72-8a46-6db1867a40c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt\" (UID: \"c9b4867c-02bd-4c72-8a46-6db1867a40c3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt"
Nov 03 20:03:21.322837 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.322824    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx4j4\" (UniqueName: \"kubernetes.io/projected/7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455-kube-api-access-fx4j4\") pod \"payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z\" (UID: \"7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z"
Nov 03 20:03:21.323338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.323318    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b4867c-02bd-4c72-8a46-6db1867a40c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt\" (UID: \"c9b4867c-02bd-4c72-8a46-6db1867a40c3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt"
Nov 03 20:03:21.339336 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.339309    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p9rh\" (UniqueName: \"kubernetes.io/projected/c9b4867c-02bd-4c72-8a46-6db1867a40c3-kube-api-access-8p9rh\") pod \"payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt\" (UID: \"c9b4867c-02bd-4c72-8a46-6db1867a40c3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt"
Nov 03 20:03:21.356809 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.356772    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="221d9a79-e987-4b03-bd4f-e1f6718fe2ce" path="/var/lib/kubelet/pods/221d9a79-e987-4b03-bd4f-e1f6718fe2ce/volumes"
Nov 03 20:03:21.357421 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.357308    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f72239c9-93c7-4f75-8296-3c8816841ac6" path="/var/lib/kubelet/pods/f72239c9-93c7-4f75-8296-3c8816841ac6/volumes"
Nov 03 20:03:21.433228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.428860    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx4j4\" (UniqueName: \"kubernetes.io/projected/7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455-kube-api-access-fx4j4\") pod \"payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z\" (UID: \"7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z"
Nov 03 20:03:21.433228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.428995    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z\" (UID: \"7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z"
Nov 03 20:03:21.433228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.429939    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z\" (UID: \"7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z"
Nov 03 20:03:21.492292 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.487890    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp" event={"ID":"e3c18563-3261-4841-81d2-3db2ec91abd3","Type":"ContainerStarted","Data":"c14f54124d5b880b893e9a8f2e0cea84cd77188b3020d6fe069fbc712f21aeec"}
Nov 03 20:03:21.492292 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.487957    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp" event={"ID":"e3c18563-3261-4841-81d2-3db2ec91abd3","Type":"ContainerStarted","Data":"f1605e8af12a8b828cc2ce9a4bfaa5f80b705a0546921c679861f060d42076a3"}
Nov 03 20:03:21.492292 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.489101    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp"
Nov 03 20:03:21.502251 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.494869    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx4j4\" (UniqueName: \"kubernetes.io/projected/7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455-kube-api-access-fx4j4\") pod \"payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z\" (UID: \"7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z"
Nov 03 20:03:21.502251 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.494952    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v" event={"ID":"26612e45-5a14-497a-9e90-06f336b9d0d1","Type":"ContainerStarted","Data":"abfd735653fa89b6e2fdf65fd9115e22269f7a36ae139c6e7f1975bbf90f1a3a"}
Nov 03 20:03:21.502251 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.495001    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v" event={"ID":"26612e45-5a14-497a-9e90-06f336b9d0d1","Type":"ContainerStarted","Data":"0471b0574acde8b43f180b30010cc163e72a928336aa5f456a91aca020f7da78"}
Nov 03 20:03:21.537257 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.528668    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt"
Nov 03 20:03:21.537257 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.529576    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp" podStartSLOduration=1.5295594399999999 podStartE2EDuration="1.52955944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:21.514766639 +0000 UTC m=+1550.287059659" watchObservedRunningTime="2026-11-03 20:03:21.52955944 +0000 UTC m=+1550.301852456"
Nov 03 20:03:21.584083 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.584023    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v" podStartSLOduration=1.584002324 podStartE2EDuration="1.584002324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:21.542950848 +0000 UTC m=+1550.315243867" watchObservedRunningTime="2026-11-03 20:03:21.584002324 +0000 UTC m=+1550.356295349"
Nov 03 20:03:21.621496 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:21.621464    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z"
Nov 03 20:03:22.033298 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.033274    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt"]
Nov 03 20:03:22.034128 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:03:22.034094    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b4867c_02bd_4c72_8a46_6db1867a40c3.slice/crio-e668bb0811bb078076f862c350947c483359042ae6199a237e361ee0aeec76b7 WatchSource:0}: Error finding container e668bb0811bb078076f862c350947c483359042ae6199a237e361ee0aeec76b7: Status 404 returned error can't find the container with id e668bb0811bb078076f862c350947c483359042ae6199a237e361ee0aeec76b7
Nov 03 20:03:22.071474 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.071444    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z"]
Nov 03 20:03:22.074710 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:03:22.074684    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3d9d49_2b92_4e5d_ba1c_f8e07ffd3455.slice/crio-fbed17e06773c6c8b795ccf1fb0d5cac272155c28a1649cd61a2b5e0d4c311c1 WatchSource:0}: Error finding container fbed17e06773c6c8b795ccf1fb0d5cac272155c28a1649cd61a2b5e0d4c311c1: Status 404 returned error can't find the container with id fbed17e06773c6c8b795ccf1fb0d5cac272155c28a1649cd61a2b5e0d4c311c1
Nov 03 20:03:22.157727 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.157699    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8"
Nov 03 20:03:22.324318 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.324230    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4"]
Nov 03 20:03:22.324559 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.324527    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4" podUID="5f124fa7-5667-4c5d-a82f-352e57290fd6" containerName="maas-api" containerID="cri-o://5bc1db6a0147cbac984ed2ef48dfd65e91fed331bdfb1192c03eee9969d61848" gracePeriod=30
Nov 03 20:03:22.504063 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.504030    5591 generic.go:356] "Generic (PLEG): container finished" podID="5f124fa7-5667-4c5d-a82f-352e57290fd6" containerID="5bc1db6a0147cbac984ed2ef48dfd65e91fed331bdfb1192c03eee9969d61848" exitCode=0
Nov 03 20:03:22.504577 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.504113    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4" event={"ID":"5f124fa7-5667-4c5d-a82f-352e57290fd6","Type":"ContainerDied","Data":"5bc1db6a0147cbac984ed2ef48dfd65e91fed331bdfb1192c03eee9969d61848"}
Nov 03 20:03:22.506420 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.506392    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt" event={"ID":"c9b4867c-02bd-4c72-8a46-6db1867a40c3","Type":"ContainerStarted","Data":"6d38f24d47ec5959d41f17a76cca7f7e4088ee4c4789f1f21a34842c36536563"}
Nov 03 20:03:22.506564 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.506446    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt" event={"ID":"c9b4867c-02bd-4c72-8a46-6db1867a40c3","Type":"ContainerStarted","Data":"e668bb0811bb078076f862c350947c483359042ae6199a237e361ee0aeec76b7"}
Nov 03 20:03:22.506656 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.506628    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt"
Nov 03 20:03:22.508418 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.508392    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z" event={"ID":"7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455","Type":"ContainerStarted","Data":"e3603e074b6bb8f82aa3597210b84d4c52c34db8d6eaf506ec72159a16bb9ca8"}
Nov 03 20:03:22.508563 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.508436    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z" event={"ID":"7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455","Type":"ContainerStarted","Data":"fbed17e06773c6c8b795ccf1fb0d5cac272155c28a1649cd61a2b5e0d4c311c1"}
Nov 03 20:03:22.508798 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.508784    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v"
Nov 03 20:03:22.528849 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.528731    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt" podStartSLOduration=1.528712039 podStartE2EDuration="1.528712039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:22.524853276 +0000 UTC m=+1551.297146298" watchObservedRunningTime="2026-11-03 20:03:22.528712039 +0000 UTC m=+1551.301005060"
Nov 03 20:03:22.548911 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.548846    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z" podStartSLOduration=1.548828519 podStartE2EDuration="1.548828519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:22.543714559 +0000 UTC m=+1551.316007578" watchObservedRunningTime="2026-11-03 20:03:22.548828519 +0000 UTC m=+1551.321121540"
Nov 03 20:03:22.631842 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.631816    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4"
Nov 03 20:03:22.785869 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.780329    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f124fa7-5667-4c5d-a82f-352e57290fd6-kube-api-access-g2tjk\" (UniqueName: \"kubernetes.io/projected/5f124fa7-5667-4c5d-a82f-352e57290fd6-kube-api-access-g2tjk\") pod \"5f124fa7-5667-4c5d-a82f-352e57290fd6\" (UID: \"5f124fa7-5667-4c5d-a82f-352e57290fd6\") "
Nov 03 20:03:22.785869 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.780417    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f124fa7-5667-4c5d-a82f-352e57290fd6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f124fa7-5667-4c5d-a82f-352e57290fd6-metrics-certs\") pod \"5f124fa7-5667-4c5d-a82f-352e57290fd6\" (UID: \"5f124fa7-5667-4c5d-a82f-352e57290fd6\") "
Nov 03 20:03:22.785869 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.780649    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f124fa7-5667-4c5d-a82f-352e57290fd6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f124fa7-5667-4c5d-a82f-352e57290fd6-openshift-service-ca\") pod \"5f124fa7-5667-4c5d-a82f-352e57290fd6\" (UID: \"5f124fa7-5667-4c5d-a82f-352e57290fd6\") "
Nov 03 20:03:22.785869 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.780716    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f124fa7-5667-4c5d-a82f-352e57290fd6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f124fa7-5667-4c5d-a82f-352e57290fd6-maas-api-tls\") pod \"5f124fa7-5667-4c5d-a82f-352e57290fd6\" (UID: \"5f124fa7-5667-4c5d-a82f-352e57290fd6\") "
Nov 03 20:03:22.785869 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.783149    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f124fa7-5667-4c5d-a82f-352e57290fd6-kube-api-access-g2tjk" pod "5f124fa7-5667-4c5d-a82f-352e57290fd6" (UID: "5f124fa7-5667-4c5d-a82f-352e57290fd6"). InnerVolumeSpecName "kube-api-access-g2tjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:03:22.785869 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.783814    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f124fa7-5667-4c5d-a82f-352e57290fd6-openshift-service-ca" pod "5f124fa7-5667-4c5d-a82f-352e57290fd6" (UID: "5f124fa7-5667-4c5d-a82f-352e57290fd6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:03:22.814508 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.809578    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2tjk\" (UniqueName: \"kubernetes.io/projected/5f124fa7-5667-4c5d-a82f-352e57290fd6-kube-api-access-g2tjk\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:22.814508 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.809614    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f124fa7-5667-4c5d-a82f-352e57290fd6-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:22.814508 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.811391    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f124fa7-5667-4c5d-a82f-352e57290fd6-metrics-certs" pod "5f124fa7-5667-4c5d-a82f-352e57290fd6" (UID: "5f124fa7-5667-4c5d-a82f-352e57290fd6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:03:22.828832 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.828772    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f124fa7-5667-4c5d-a82f-352e57290fd6-maas-api-tls" pod "5f124fa7-5667-4c5d-a82f-352e57290fd6" (UID: "5f124fa7-5667-4c5d-a82f-352e57290fd6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:03:22.915299 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.914181    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f124fa7-5667-4c5d-a82f-352e57290fd6-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:22.915299 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:22.914235    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f124fa7-5667-4c5d-a82f-352e57290fd6-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:03:23.516534 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:23.516505    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4"
Nov 03 20:03:23.517025 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:23.516539    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4" event={"ID":"5f124fa7-5667-4c5d-a82f-352e57290fd6","Type":"ContainerDied","Data":"a1b535d9a23fff2094dc160cb4040ef444baeacacee980657399d9a04205e0c8"}
Nov 03 20:03:23.517025 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:23.516601    5591 scope.go:122] "RemoveContainer" containerID="5bc1db6a0147cbac984ed2ef48dfd65e91fed331bdfb1192c03eee9969d61848"
Nov 03 20:03:23.517152 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:23.517068    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z"
Nov 03 20:03:23.536683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:23.536653    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4"]
Nov 03 20:03:23.540231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:23.540190    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c131fa-d694fcc5c-mlnc4"]
Nov 03 20:03:25.354851 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:25.354815    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f124fa7-5667-4c5d-a82f-352e57290fd6" path="/var/lib/kubelet/pods/5f124fa7-5667-4c5d-a82f-352e57290fd6/volumes"
Nov 03 20:03:26.074573 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:26.063677    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n"]
Nov 03 20:03:26.074573 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:26.064786    5591 cpu_manager.go:414] "containerMap: removing container" podUID="5f124fa7-5667-4c5d-a82f-352e57290fd6" containerName="maas-api"
Nov 03 20:03:26.074573 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:26.064807    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f124fa7-5667-4c5d-a82f-352e57290fd6" containerName="maas-api"
Nov 03 20:03:26.074573 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:26.065061    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f124fa7-5667-4c5d-a82f-352e57290fd6" containerName="maas-api"
Nov 03 20:03:26.074573 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:26.071964    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n"
Nov 03 20:03:26.077302 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:26.077272    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-bfc240-serving-cert\""
Nov 03 20:03:26.080073 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:26.079928    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n"]
Nov 03 20:03:26.210229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:26.210033    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n\" (UID: \"9d7c8f87-8009-4efa-a92f-3dd314f7bac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n"
Nov 03 20:03:26.210229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:26.210136    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n\" (UID: \"9d7c8f87-8009-4efa-a92f-3dd314f7bac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n"
Nov 03 20:03:26.210497 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:26.210246    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n\" (UID: \"9d7c8f87-8009-4efa-a92f-3dd314f7bac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n"
Nov 03 20:03:26.210497 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:26.210367    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz2gw\" (UniqueName: \"kubernetes.io/projected/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-kube-api-access-pz2gw\") pod \"maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n\" (UID: \"9d7c8f87-8009-4efa-a92f-3dd314f7bac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n"
Nov 03 20:03:26.311593 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:26.311559    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz2gw\" (UniqueName: \"kubernetes.io/projected/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-kube-api-access-pz2gw\") pod \"maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n\" (UID: \"9d7c8f87-8009-4efa-a92f-3dd314f7bac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n"
Nov 03 20:03:26.311785 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:26.311668    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n\" (UID: \"9d7c8f87-8009-4efa-a92f-3dd314f7bac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n"
Nov 03 20:03:26.311785 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:26.311739    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n\" (UID: \"9d7c8f87-8009-4efa-a92f-3dd314f7bac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n"
Nov 03 20:03:26.311904 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:26.311824    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n\" (UID: \"9d7c8f87-8009-4efa-a92f-3dd314f7bac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n"
Nov 03 20:03:26.312478 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:26.312455    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n\" (UID: \"9d7c8f87-8009-4efa-a92f-3dd314f7bac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n"
Nov 03 20:03:26.314558 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:26.314524    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n\" (UID: \"9d7c8f87-8009-4efa-a92f-3dd314f7bac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n"
Nov 03 20:03:26.314558 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:26.314546    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n\" (UID: \"9d7c8f87-8009-4efa-a92f-3dd314f7bac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n"
Nov 03 20:03:26.323885 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:26.323864    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz2gw\" (UniqueName: \"kubernetes.io/projected/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-kube-api-access-pz2gw\") pod \"maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n\" (UID: \"9d7c8f87-8009-4efa-a92f-3dd314f7bac0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n"
Nov 03 20:03:26.400480 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:26.400403    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n"
Nov 03 20:03:26.689013 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:26.688986    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n"]
Nov 03 20:03:27.588062 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:27.588029    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n" event={"ID":"9d7c8f87-8009-4efa-a92f-3dd314f7bac0","Type":"ContainerStarted","Data":"72aa9d5a68cb141b51757eed21f00436a942681092448f149bf2bdec38a2e8f7"}
Nov 03 20:03:27.588698 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:27.588090    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n" event={"ID":"9d7c8f87-8009-4efa-a92f-3dd314f7bac0","Type":"ContainerStarted","Data":"afbb114bebb59384e355d1e8c4b969d0957b5a80d2ce5a73096c8c7859c25632"}
Nov 03 20:03:27.589141 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:27.589118    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n"
Nov 03 20:03:27.618515 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:27.618468    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n" podStartSLOduration=1.230306782 podStartE2EDuration="1.618451061s" totalImagesPullingTime="388.144279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:27.616360398 +0000 UTC m=+1556.388653414" watchObservedRunningTime="2026-11-03 20:03:27.618451061 +0000 UTC m=+1556.390744079"
Nov 03 20:03:28.727798 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:03:28.727760    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:03:29.050455 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:29.050418    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h"]
Nov 03 20:03:29.076815 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:29.076749    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h"]
Nov 03 20:03:29.077054 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:29.076912    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h"
Nov 03 20:03:29.088544 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:29.088511    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-f154ac-serving-cert\""
Nov 03 20:03:29.089485 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:29.089454    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1031c5e-2ece-4213-ba23-33126946df5a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h\" (UID: \"d1031c5e-2ece-4213-ba23-33126946df5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h"
Nov 03 20:03:29.089699 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:29.089672    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1031c5e-2ece-4213-ba23-33126946df5a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h\" (UID: \"d1031c5e-2ece-4213-ba23-33126946df5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h"
Nov 03 20:03:29.089776 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:29.089711    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1031c5e-2ece-4213-ba23-33126946df5a-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h\" (UID: \"d1031c5e-2ece-4213-ba23-33126946df5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h"
Nov 03 20:03:29.089915 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:29.089897    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7mfx\" (UniqueName: \"kubernetes.io/projected/d1031c5e-2ece-4213-ba23-33126946df5a-kube-api-access-d7mfx\") pod \"maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h\" (UID: \"d1031c5e-2ece-4213-ba23-33126946df5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h"
Nov 03 20:03:29.194245 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:29.191190    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7mfx\" (UniqueName: \"kubernetes.io/projected/d1031c5e-2ece-4213-ba23-33126946df5a-kube-api-access-d7mfx\") pod \"maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h\" (UID: \"d1031c5e-2ece-4213-ba23-33126946df5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h"
Nov 03 20:03:29.194245 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:29.191384    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1031c5e-2ece-4213-ba23-33126946df5a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h\" (UID: \"d1031c5e-2ece-4213-ba23-33126946df5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h"
Nov 03 20:03:29.194245 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:29.191508    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1031c5e-2ece-4213-ba23-33126946df5a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h\" (UID: \"d1031c5e-2ece-4213-ba23-33126946df5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h"
Nov 03 20:03:29.194245 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:29.191536    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1031c5e-2ece-4213-ba23-33126946df5a-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h\" (UID: \"d1031c5e-2ece-4213-ba23-33126946df5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h"
Nov 03 20:03:29.230428 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:29.215999    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1031c5e-2ece-4213-ba23-33126946df5a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h\" (UID: \"d1031c5e-2ece-4213-ba23-33126946df5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h"
Nov 03 20:03:29.230789 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:29.226864    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1031c5e-2ece-4213-ba23-33126946df5a-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h\" (UID: \"d1031c5e-2ece-4213-ba23-33126946df5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h"
Nov 03 20:03:29.254801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:29.254766    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1031c5e-2ece-4213-ba23-33126946df5a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h\" (UID: \"d1031c5e-2ece-4213-ba23-33126946df5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h"
Nov 03 20:03:29.262949 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:29.262913    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7mfx\" (UniqueName: \"kubernetes.io/projected/d1031c5e-2ece-4213-ba23-33126946df5a-kube-api-access-d7mfx\") pod \"maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h\" (UID: \"d1031c5e-2ece-4213-ba23-33126946df5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h"
Nov 03 20:03:29.409729 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:29.409647    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h"
Nov 03 20:03:29.697583 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:29.697498    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h"]
Nov 03 20:03:29.702432 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:03:29.702402    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1031c5e_2ece_4213_ba23_33126946df5a.slice/crio-033cfedb9cf8d8889f800fb09b2f37a29fce577128987462e413343d4925914a WatchSource:0}: Error finding container 033cfedb9cf8d8889f800fb09b2f37a29fce577128987462e413343d4925914a: Status 404 returned error can't find the container with id 033cfedb9cf8d8889f800fb09b2f37a29fce577128987462e413343d4925914a
Nov 03 20:03:30.612803 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:30.612754    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h" event={"ID":"d1031c5e-2ece-4213-ba23-33126946df5a","Type":"ContainerStarted","Data":"b2b2ea9a9df5580537698cb2fdfb2917a47d849d546f61807ece727563d98317"}
Nov 03 20:03:30.613292 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:30.612835    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h" event={"ID":"d1031c5e-2ece-4213-ba23-33126946df5a","Type":"ContainerStarted","Data":"033cfedb9cf8d8889f800fb09b2f37a29fce577128987462e413343d4925914a"}
Nov 03 20:03:30.613292 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:30.612949    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h"
Nov 03 20:03:30.634953 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:30.634893    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h" podStartSLOduration=1.15019494 podStartE2EDuration="1.634874773s" totalImagesPullingTime="484.679833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:30.634056 +0000 UTC m=+1559.406349020" watchObservedRunningTime="2026-11-03 20:03:30.634874773 +0000 UTC m=+1559.407167792"
Nov 03 20:03:31.242896 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:31.242861    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:03:32.345693 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:32.345657    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc"]
Nov 03 20:03:32.351517 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:32.351495    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm"]
Nov 03 20:03:32.351659 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:32.351609    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc"
Nov 03 20:03:32.356113 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:32.356092    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm"
Nov 03 20:03:32.361592 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:32.361567    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-81a5ae\""
Nov 03 20:03:32.364709 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:32.364686    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc"]
Nov 03 20:03:32.368393 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:32.367579    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-81a5ae-dockercfg-4scdm\""
Nov 03 20:03:32.370517 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:32.370493    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm"]
Nov 03 20:03:32.501726 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:32.501692    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/150e0fa8-4c32-418d-ae65-384d1baf47b9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm\" (UID: \"150e0fa8-4c32-418d-ae65-384d1baf47b9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm"
Nov 03 20:03:32.501913 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:32.501872    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l6zp\" (UniqueName: \"kubernetes.io/projected/150e0fa8-4c32-418d-ae65-384d1baf47b9-kube-api-access-7l6zp\") pod \"payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm\" (UID: \"150e0fa8-4c32-418d-ae65-384d1baf47b9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm"
Nov 03 20:03:32.501958 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:32.501916    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67ada42-276d-41e3-bb13-05a8bd3d1029-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc\" (UID: \"d67ada42-276d-41e3-bb13-05a8bd3d1029\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc"
Nov 03 20:03:32.502052 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:32.502037    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lxbb\" (UniqueName: \"kubernetes.io/projected/d67ada42-276d-41e3-bb13-05a8bd3d1029-kube-api-access-7lxbb\") pod \"payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc\" (UID: \"d67ada42-276d-41e3-bb13-05a8bd3d1029\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc"
Nov 03 20:03:32.609289 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:32.603523    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/150e0fa8-4c32-418d-ae65-384d1baf47b9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm\" (UID: \"150e0fa8-4c32-418d-ae65-384d1baf47b9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm"
Nov 03 20:03:32.609289 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:32.603727    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l6zp\" (UniqueName: \"kubernetes.io/projected/150e0fa8-4c32-418d-ae65-384d1baf47b9-kube-api-access-7l6zp\") pod \"payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm\" (UID: \"150e0fa8-4c32-418d-ae65-384d1baf47b9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm"
Nov 03 20:03:32.609289 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:32.603805    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67ada42-276d-41e3-bb13-05a8bd3d1029-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc\" (UID: \"d67ada42-276d-41e3-bb13-05a8bd3d1029\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc"
Nov 03 20:03:32.609289 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:32.603986    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lxbb\" (UniqueName: \"kubernetes.io/projected/d67ada42-276d-41e3-bb13-05a8bd3d1029-kube-api-access-7lxbb\") pod \"payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc\" (UID: \"d67ada42-276d-41e3-bb13-05a8bd3d1029\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc"
Nov 03 20:03:32.609289 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:32.605378    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/150e0fa8-4c32-418d-ae65-384d1baf47b9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm\" (UID: \"150e0fa8-4c32-418d-ae65-384d1baf47b9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm"
Nov 03 20:03:32.622704 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:32.622671    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67ada42-276d-41e3-bb13-05a8bd3d1029-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc\" (UID: \"d67ada42-276d-41e3-bb13-05a8bd3d1029\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc"
Nov 03 20:03:32.628132 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:32.628093    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lxbb\" (UniqueName: \"kubernetes.io/projected/d67ada42-276d-41e3-bb13-05a8bd3d1029-kube-api-access-7lxbb\") pod \"payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc\" (UID: \"d67ada42-276d-41e3-bb13-05a8bd3d1029\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc"
Nov 03 20:03:32.682258 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:32.660293    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l6zp\" (UniqueName: \"kubernetes.io/projected/150e0fa8-4c32-418d-ae65-384d1baf47b9-kube-api-access-7l6zp\") pod \"payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm\" (UID: \"150e0fa8-4c32-418d-ae65-384d1baf47b9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm"
Nov 03 20:03:32.682258 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:32.672906    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc"
Nov 03 20:03:32.736536 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:32.736253    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm"
Nov 03 20:03:33.579298 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:33.578066    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp"
Nov 03 20:03:33.580193 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:33.579942    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt"
Nov 03 20:03:33.604482 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:33.582351    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v"
Nov 03 20:03:33.998421 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:33.992355    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:34.272479 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:34.115292    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 20:03:34.272479 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:34.192938    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 20:03:34.305314 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:34.304333    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:34.306167 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:34.306098    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:34.419249 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:34.391453    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:34.691342 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:34.673412    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:34.771237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:34.737488    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded"
Nov 03 20:03:34.824234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:34.739503    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:34.970405 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:34.970310    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm"]
Nov 03 20:03:35.033074 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:34.998415    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:35.044233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:35.034891    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z"
Nov 03 20:03:35.257316 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:35.234708    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:35.257316 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:35.243954    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 20:03:35.257316 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:35.244018    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n"
Nov 03 20:03:35.257316 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:35.244067    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 20:03:35.342644 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:35.287346    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:35.342644 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:35.328365    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:03:35.342644 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:35.330413    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:35.342644 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:35.334583    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 20:03:35.342644 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:35.334641    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 20:03:35.342644 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:35.334677    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 20:03:35.410503 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:35.410453    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:35.518653 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:35.518332    5591 patch_prober.go:29] interesting pod/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:03:35.518653 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:35.518390    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n" podUID="2b1c87e7-ecf0-438f-a3c4-93da53813516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:03:35.717548 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:35.717495    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc"]
Nov 03 20:03:35.725588 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:35.725555    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 20:03:35.726314 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:35.726292    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 20:03:35.774366 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:35.748537    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 20:03:35.774366 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:35.748655    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 20:03:35.774366 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:35.748706    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 20:03:35.774366 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:35.748827    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 20:03:35.861918 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:35.861883    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 20:03:36.628589 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:36.628559    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h"
Nov 03 20:03:36.719493 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:36.719457    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm" event={"ID":"150e0fa8-4c32-418d-ae65-384d1baf47b9","Type":"ContainerStarted","Data":"027a82223e775ceaf8f525069da2e28ecb42247d963bd82aa5c707eda4fe870c"}
Nov 03 20:03:36.719986 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:36.719536    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm" event={"ID":"150e0fa8-4c32-418d-ae65-384d1baf47b9","Type":"ContainerStarted","Data":"3ba13573d7804eec68435d85e9a32248329a9ca6b6ddc6ff687d8fcd5f323f80"}
Nov 03 20:03:36.719986 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:36.719592    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm"
Nov 03 20:03:36.722447 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:36.722401    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc" event={"ID":"d67ada42-276d-41e3-bb13-05a8bd3d1029","Type":"ContainerStarted","Data":"712946f2423f14f6ea086b6f037db4354d3a0893015cb4c1a4a87366306b8e48"}
Nov 03 20:03:36.722593 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:36.722498    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc"
Nov 03 20:03:36.722593 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:36.722526    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc" event={"ID":"d67ada42-276d-41e3-bb13-05a8bd3d1029","Type":"ContainerStarted","Data":"b74131792ba217a0bfe82d5e349d53055f2941a09e1ef5979822479ef07b3b85"}
Nov 03 20:03:36.743801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:36.743745    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm" podStartSLOduration=4.743730075 podStartE2EDuration="4.743730075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:36.740157187 +0000 UTC m=+1565.512450206" watchObservedRunningTime="2026-11-03 20:03:36.743730075 +0000 UTC m=+1565.516023136"
Nov 03 20:03:36.761607 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:36.761531    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc" podStartSLOduration=4.761511721 podStartE2EDuration="4.761511721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:36.758286178 +0000 UTC m=+1565.530579197" watchObservedRunningTime="2026-11-03 20:03:36.761511721 +0000 UTC m=+1565.533804739"
Nov 03 20:03:40.235257 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:40.235219    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4"]
Nov 03 20:03:40.241381 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:40.241355    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4"
Nov 03 20:03:40.247448 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:40.247415    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-88a18e\""
Nov 03 20:03:40.247615 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:40.247554    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-88a18e-dockercfg-89lxf\""
Nov 03 20:03:40.251817 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:40.250808    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4"]
Nov 03 20:03:40.274822 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:40.274796    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp"]
Nov 03 20:03:40.283676 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:40.283638    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp"
Nov 03 20:03:40.300723 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:40.300508    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp"]
Nov 03 20:03:40.331765 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:40.331719    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glhsd\" (UniqueName: \"kubernetes.io/projected/a1cd929e-9b0d-4903-a019-af1f5893c5db-kube-api-access-glhsd\") pod \"payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4\" (UID: \"a1cd929e-9b0d-4903-a019-af1f5893c5db\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4"
Nov 03 20:03:40.331968 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:40.331835    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1cd929e-9b0d-4903-a019-af1f5893c5db-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4\" (UID: \"a1cd929e-9b0d-4903-a019-af1f5893c5db\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4"
Nov 03 20:03:40.448235 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:40.436185    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1cd929e-9b0d-4903-a019-af1f5893c5db-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4\" (UID: \"a1cd929e-9b0d-4903-a019-af1f5893c5db\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4"
Nov 03 20:03:40.448235 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:40.436290    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1235dbc4-10ee-4c05-b723-49bab294b390-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp\" (UID: \"1235dbc4-10ee-4c05-b723-49bab294b390\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp"
Nov 03 20:03:40.448235 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:40.436423    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k8sd\" (UniqueName: \"kubernetes.io/projected/1235dbc4-10ee-4c05-b723-49bab294b390-kube-api-access-7k8sd\") pod \"payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp\" (UID: \"1235dbc4-10ee-4c05-b723-49bab294b390\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp"
Nov 03 20:03:40.448235 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:40.436614    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glhsd\" (UniqueName: \"kubernetes.io/projected/a1cd929e-9b0d-4903-a019-af1f5893c5db-kube-api-access-glhsd\") pod \"payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4\" (UID: \"a1cd929e-9b0d-4903-a019-af1f5893c5db\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4"
Nov 03 20:03:40.448235 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:40.437879    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1cd929e-9b0d-4903-a019-af1f5893c5db-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4\" (UID: \"a1cd929e-9b0d-4903-a019-af1f5893c5db\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4"
Nov 03 20:03:40.469142 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:40.466965    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glhsd\" (UniqueName: \"kubernetes.io/projected/a1cd929e-9b0d-4903-a019-af1f5893c5db-kube-api-access-glhsd\") pod \"payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4\" (UID: \"a1cd929e-9b0d-4903-a019-af1f5893c5db\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4"
Nov 03 20:03:40.544181 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:40.544124    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1235dbc4-10ee-4c05-b723-49bab294b390-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp\" (UID: \"1235dbc4-10ee-4c05-b723-49bab294b390\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp"
Nov 03 20:03:40.545150 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:40.545121    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1235dbc4-10ee-4c05-b723-49bab294b390-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp\" (UID: \"1235dbc4-10ee-4c05-b723-49bab294b390\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp"
Nov 03 20:03:40.545406 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:40.545384    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k8sd\" (UniqueName: \"kubernetes.io/projected/1235dbc4-10ee-4c05-b723-49bab294b390-kube-api-access-7k8sd\") pod \"payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp\" (UID: \"1235dbc4-10ee-4c05-b723-49bab294b390\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp"
Nov 03 20:03:40.565994 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:40.559148    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4"
Nov 03 20:03:40.580234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:40.579941    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k8sd\" (UniqueName: \"kubernetes.io/projected/1235dbc4-10ee-4c05-b723-49bab294b390-kube-api-access-7k8sd\") pod \"payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp\" (UID: \"1235dbc4-10ee-4c05-b723-49bab294b390\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp"
Nov 03 20:03:40.628617 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:40.622590    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp"
Nov 03 20:03:40.952981 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:03:40.952952    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1cd929e_9b0d_4903_a019_af1f5893c5db.slice/crio-ec41cc1c4fba938f0eff8cc983362f3eafaa04f183c58cbefda6c7db92329378 WatchSource:0}: Error finding container ec41cc1c4fba938f0eff8cc983362f3eafaa04f183c58cbefda6c7db92329378: Status 404 returned error can't find the container with id ec41cc1c4fba938f0eff8cc983362f3eafaa04f183c58cbefda6c7db92329378
Nov 03 20:03:40.953715 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:40.953673    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4"]
Nov 03 20:03:40.959590 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:03:40.959561    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1235dbc4_10ee_4c05_b723_49bab294b390.slice/crio-26606330886a296dd390c972b14d0eb59e3e902dd31130769ccd002c03525de2 WatchSource:0}: Error finding container 26606330886a296dd390c972b14d0eb59e3e902dd31130769ccd002c03525de2: Status 404 returned error can't find the container with id 26606330886a296dd390c972b14d0eb59e3e902dd31130769ccd002c03525de2
Nov 03 20:03:40.959724 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:40.959702    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp"]
Nov 03 20:03:41.851828 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:41.851462    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp" event={"ID":"1235dbc4-10ee-4c05-b723-49bab294b390","Type":"ContainerStarted","Data":"cb3419aa7659a4d3f99bd58f6adb80f6321ce1acdb4d3faf33fcfcfc2e4d0147"}
Nov 03 20:03:41.851828 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:41.851769    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp"
Nov 03 20:03:41.851828 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:41.851804    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp" event={"ID":"1235dbc4-10ee-4c05-b723-49bab294b390","Type":"ContainerStarted","Data":"26606330886a296dd390c972b14d0eb59e3e902dd31130769ccd002c03525de2"}
Nov 03 20:03:41.853473 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:41.853437    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4" event={"ID":"a1cd929e-9b0d-4903-a019-af1f5893c5db","Type":"ContainerStarted","Data":"5e9311bb1019881692967d01d57faf62d326bc64a4cd03edb7d226054c9dd84d"}
Nov 03 20:03:41.853634 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:41.853494    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4" event={"ID":"a1cd929e-9b0d-4903-a019-af1f5893c5db","Type":"ContainerStarted","Data":"ec41cc1c4fba938f0eff8cc983362f3eafaa04f183c58cbefda6c7db92329378"}
Nov 03 20:03:41.854131 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:41.854108    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4"
Nov 03 20:03:41.871965 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:41.871903    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp" podStartSLOduration=1.871887726 podStartE2EDuration="1.871887726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:41.869023912 +0000 UTC m=+1570.641316941" watchObservedRunningTime="2026-11-03 20:03:41.871887726 +0000 UTC m=+1570.644180765"
Nov 03 20:03:41.888678 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:41.888177    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4" podStartSLOduration=1.888157599 podStartE2EDuration="1.888157599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:41.887970054 +0000 UTC m=+1570.660263073" watchObservedRunningTime="2026-11-03 20:03:41.888157599 +0000 UTC m=+1570.660450618"
Nov 03 20:03:42.930382 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:42.930347    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6"]
Nov 03 20:03:42.936843 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:42.936809    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6"
Nov 03 20:03:42.949099 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:42.945458    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-81a5ae-serving-cert\""
Nov 03 20:03:42.949099 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:42.946417    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6"]
Nov 03 20:03:43.129409 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:43.129367    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4ae1ca-060a-43e4-b877-02a2c3565899-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6\" (UID: \"9a4ae1ca-060a-43e4-b877-02a2c3565899\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6"
Nov 03 20:03:43.129752 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:43.129726    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a4ae1ca-060a-43e4-b877-02a2c3565899-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6\" (UID: \"9a4ae1ca-060a-43e4-b877-02a2c3565899\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6"
Nov 03 20:03:43.130061 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:43.130042    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4ae1ca-060a-43e4-b877-02a2c3565899-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6\" (UID: \"9a4ae1ca-060a-43e4-b877-02a2c3565899\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6"
Nov 03 20:03:43.130342 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:43.130323    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvj8c\" (UniqueName: \"kubernetes.io/projected/9a4ae1ca-060a-43e4-b877-02a2c3565899-kube-api-access-zvj8c\") pod \"maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6\" (UID: \"9a4ae1ca-060a-43e4-b877-02a2c3565899\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6"
Nov 03 20:03:43.231576 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:43.231486    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4ae1ca-060a-43e4-b877-02a2c3565899-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6\" (UID: \"9a4ae1ca-060a-43e4-b877-02a2c3565899\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6"
Nov 03 20:03:43.231735 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:43.231622    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvj8c\" (UniqueName: \"kubernetes.io/projected/9a4ae1ca-060a-43e4-b877-02a2c3565899-kube-api-access-zvj8c\") pod \"maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6\" (UID: \"9a4ae1ca-060a-43e4-b877-02a2c3565899\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6"
Nov 03 20:03:43.231789 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:43.231757    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4ae1ca-060a-43e4-b877-02a2c3565899-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6\" (UID: \"9a4ae1ca-060a-43e4-b877-02a2c3565899\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6"
Nov 03 20:03:43.231850 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:43.231821    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a4ae1ca-060a-43e4-b877-02a2c3565899-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6\" (UID: \"9a4ae1ca-060a-43e4-b877-02a2c3565899\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6"
Nov 03 20:03:43.232759 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:43.232673    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4ae1ca-060a-43e4-b877-02a2c3565899-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6\" (UID: \"9a4ae1ca-060a-43e4-b877-02a2c3565899\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6"
Nov 03 20:03:43.234584 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:43.234553    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4ae1ca-060a-43e4-b877-02a2c3565899-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6\" (UID: \"9a4ae1ca-060a-43e4-b877-02a2c3565899\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6"
Nov 03 20:03:43.234714 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:43.234608    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a4ae1ca-060a-43e4-b877-02a2c3565899-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6\" (UID: \"9a4ae1ca-060a-43e4-b877-02a2c3565899\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6"
Nov 03 20:03:43.243590 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:43.243564    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvj8c\" (UniqueName: \"kubernetes.io/projected/9a4ae1ca-060a-43e4-b877-02a2c3565899-kube-api-access-zvj8c\") pod \"maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6\" (UID: \"9a4ae1ca-060a-43e4-b877-02a2c3565899\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6"
Nov 03 20:03:43.284591 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:43.276508    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6"
Nov 03 20:03:43.715579 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:43.715547    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6"]
Nov 03 20:03:43.718696 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:03:43.718654    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a4ae1ca_060a_43e4_b877_02a2c3565899.slice/crio-569ea883e14c0aeb2a2d0c55216068a476d83ed03fe521eea2202ee826d30860 WatchSource:0}: Error finding container 569ea883e14c0aeb2a2d0c55216068a476d83ed03fe521eea2202ee826d30860: Status 404 returned error can't find the container with id 569ea883e14c0aeb2a2d0c55216068a476d83ed03fe521eea2202ee826d30860
Nov 03 20:03:43.874419 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:43.874365    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6" event={"ID":"9a4ae1ca-060a-43e4-b877-02a2c3565899","Type":"ContainerStarted","Data":"569ea883e14c0aeb2a2d0c55216068a476d83ed03fe521eea2202ee826d30860"}
Nov 03 20:03:44.883987 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:44.883942    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6" event={"ID":"9a4ae1ca-060a-43e4-b877-02a2c3565899","Type":"ContainerStarted","Data":"1359ce024772bfee9df5dc69fcf03c0a8aeaba076480724d339a894e23a8a5cf"}
Nov 03 20:03:44.884435 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:44.884107    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6"
Nov 03 20:03:44.909164 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:44.909104    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6" podStartSLOduration=2.231325992 podStartE2EDuration="2.909084583s" totalImagesPullingTime="677.758591ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:44.905455061 +0000 UTC m=+1573.677748081" watchObservedRunningTime="2026-11-03 20:03:44.909084583 +0000 UTC m=+1573.681377603"
Nov 03 20:03:45.630106 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:45.628330    5591 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": read tcp 10.134.0.2:53878->10.134.0.161:15021: read: connection reset by peer" start-of-body=
Nov 03 20:03:45.630106 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:45.628396    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f" podUID="7aff0e54-0f19-41b1-af95-70aee78fea74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": read tcp 10.134.0.2:53878->10.134.0.161:15021: read: connection reset by peer"
Nov 03 20:03:45.933577 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:45.932435    5591 generic.go:356] "Generic (PLEG): container finished" podID="7aff0e54-0f19-41b1-af95-70aee78fea74" containerID="519e0c1283e7fc13fbf46f1e602664f635ae39749f0aae6bbb1f164e4a7f884f" exitCode=0
Nov 03 20:03:45.933577 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:45.932517    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f" event={"ID":"7aff0e54-0f19-41b1-af95-70aee78fea74","Type":"ContainerDied","Data":"519e0c1283e7fc13fbf46f1e602664f635ae39749f0aae6bbb1f164e4a7f884f"}
Nov 03 20:03:45.935107 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:45.935078    5591 scope.go:122] "RemoveContainer" containerID="519e0c1283e7fc13fbf46f1e602664f635ae39749f0aae6bbb1f164e4a7f884f"
Nov 03 20:03:45.937106 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:45.937071    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:03:45.937275 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:45.937161    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:03:45.937275 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:45.937230    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:03:46.945009 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:46.944972    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f" event={"ID":"7aff0e54-0f19-41b1-af95-70aee78fea74","Type":"ContainerStarted","Data":"87b15fa1df7d5891154e206f57ada0197c88be773ef7beb228a0eda2465bca27"}
Nov 03 20:03:47.094615 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:47.094581    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:03:47.095002 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:47.094912    5591 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 03 20:03:47.095152 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:47.095129    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f" podUID="7aff0e54-0f19-41b1-af95-70aee78fea74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 03 20:03:47.805399 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:47.805037    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc"
Nov 03 20:03:47.807546 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:47.807523    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm"
Nov 03 20:03:48.097288 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:48.096640    5591 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 03 20:03:48.097288 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:48.096707    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f" podUID="7aff0e54-0f19-41b1-af95-70aee78fea74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 03 20:03:48.841444 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:48.841411    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc"]
Nov 03 20:03:48.846954 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:48.846933    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc"
Nov 03 20:03:48.852560 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:48.852535    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-bcfde8\""
Nov 03 20:03:48.852732 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:48.852571    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-bcfde8-dockercfg-nskhm\""
Nov 03 20:03:48.857373 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:48.857351    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc"]
Nov 03 20:03:48.910235 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:48.901230    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr"]
Nov 03 20:03:48.914384 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:48.914357    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr"
Nov 03 20:03:48.915027 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:48.914974    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr"]
Nov 03 20:03:48.917769 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:48.917737    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85b7cb7-d242-4015-8bd9-bd6a583fae15-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc\" (UID: \"c85b7cb7-d242-4015-8bd9-bd6a583fae15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc"
Nov 03 20:03:48.917907 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:48.917799    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsf7v\" (UniqueName: \"kubernetes.io/projected/c85b7cb7-d242-4015-8bd9-bd6a583fae15-kube-api-access-lsf7v\") pod \"payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc\" (UID: \"c85b7cb7-d242-4015-8bd9-bd6a583fae15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc"
Nov 03 20:03:49.092331 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:49.092237    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85b7cb7-d242-4015-8bd9-bd6a583fae15-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc\" (UID: \"c85b7cb7-d242-4015-8bd9-bd6a583fae15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc"
Nov 03 20:03:49.092639 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:49.092614    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsf7v\" (UniqueName: \"kubernetes.io/projected/c85b7cb7-d242-4015-8bd9-bd6a583fae15-kube-api-access-lsf7v\") pod \"payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc\" (UID: \"c85b7cb7-d242-4015-8bd9-bd6a583fae15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc"
Nov 03 20:03:49.092849 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:49.092831    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh6kr\" (UniqueName: \"kubernetes.io/projected/122f235f-b27f-4342-8f25-f004502a0291-kube-api-access-jh6kr\") pod \"payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr\" (UID: \"122f235f-b27f-4342-8f25-f004502a0291\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr"
Nov 03 20:03:49.093349 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:49.093322    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122f235f-b27f-4342-8f25-f004502a0291-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr\" (UID: \"122f235f-b27f-4342-8f25-f004502a0291\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr"
Nov 03 20:03:49.098230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:49.094165    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85b7cb7-d242-4015-8bd9-bd6a583fae15-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc\" (UID: \"c85b7cb7-d242-4015-8bd9-bd6a583fae15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc"
Nov 03 20:03:49.132227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:49.111450    5591 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 03 20:03:49.132227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:49.111539    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f" podUID="7aff0e54-0f19-41b1-af95-70aee78fea74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 03 20:03:49.146609 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:49.142918    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsf7v\" (UniqueName: \"kubernetes.io/projected/c85b7cb7-d242-4015-8bd9-bd6a583fae15-kube-api-access-lsf7v\") pod \"payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc\" (UID: \"c85b7cb7-d242-4015-8bd9-bd6a583fae15\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc"
Nov 03 20:03:49.169684 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:49.169640    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc"
Nov 03 20:03:49.221963 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:49.221921    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122f235f-b27f-4342-8f25-f004502a0291-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr\" (UID: \"122f235f-b27f-4342-8f25-f004502a0291\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr"
Nov 03 20:03:49.228650 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:49.227738    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122f235f-b27f-4342-8f25-f004502a0291-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr\" (UID: \"122f235f-b27f-4342-8f25-f004502a0291\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr"
Nov 03 20:03:49.228650 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:49.227902    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh6kr\" (UniqueName: \"kubernetes.io/projected/122f235f-b27f-4342-8f25-f004502a0291-kube-api-access-jh6kr\") pod \"payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr\" (UID: \"122f235f-b27f-4342-8f25-f004502a0291\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr"
Nov 03 20:03:49.297233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:49.294952    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh6kr\" (UniqueName: \"kubernetes.io/projected/122f235f-b27f-4342-8f25-f004502a0291-kube-api-access-jh6kr\") pod \"payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr\" (UID: \"122f235f-b27f-4342-8f25-f004502a0291\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr"
Nov 03 20:03:49.530273 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:49.530145    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr"
Nov 03 20:03:49.610803 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:49.610700    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc"]
Nov 03 20:03:49.626294 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:03:49.626255    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85b7cb7_d242_4015_8bd9_bd6a583fae15.slice/crio-ea450e875dab50413e7ad3749c02565b306a9b5093dab2d1df5ed5a86a5334e0 WatchSource:0}: Error finding container ea450e875dab50413e7ad3749c02565b306a9b5093dab2d1df5ed5a86a5334e0: Status 404 returned error can't find the container with id ea450e875dab50413e7ad3749c02565b306a9b5093dab2d1df5ed5a86a5334e0
Nov 03 20:03:49.964111 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:49.961785    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr"]
Nov 03 20:03:50.101232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:50.100677    5591 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 03 20:03:50.101232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:50.100744    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f" podUID="7aff0e54-0f19-41b1-af95-70aee78fea74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 03 20:03:50.111420 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:50.111393    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr" event={"ID":"122f235f-b27f-4342-8f25-f004502a0291","Type":"ContainerStarted","Data":"d11ef053c5d94301ee9b6b7def451f0ce15af01d5e03c06d5b4fde65db512dbc"}
Nov 03 20:03:50.113521 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:50.113494    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc" event={"ID":"c85b7cb7-d242-4015-8bd9-bd6a583fae15","Type":"ContainerStarted","Data":"073b8b2fc768b07b20f218f3edb204a02c33a489bb41c7449071c989ebd991ae"}
Nov 03 20:03:50.113670 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:50.113543    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc" event={"ID":"c85b7cb7-d242-4015-8bd9-bd6a583fae15","Type":"ContainerStarted","Data":"ea450e875dab50413e7ad3749c02565b306a9b5093dab2d1df5ed5a86a5334e0"}
Nov 03 20:03:50.114237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:50.114169    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc"
Nov 03 20:03:50.134461 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:50.134412    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc" podStartSLOduration=2.134398076 podStartE2EDuration="2.134398076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:50.134321636 +0000 UTC m=+1578.906614651" watchObservedRunningTime="2026-11-03 20:03:50.134398076 +0000 UTC m=+1578.906691091"
Nov 03 20:03:50.310649 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:50.310609    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2"]
Nov 03 20:03:50.317469 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:50.317438    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2"]
Nov 03 20:03:50.317662 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:50.317556    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2"
Nov 03 20:03:50.323634 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:50.323606    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-43c674\""
Nov 03 20:03:50.353497 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:50.353458    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d9d77f5-8819-447e-b0a0-c27ce2d1d498-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2\" (UID: \"9d9d77f5-8819-447e-b0a0-c27ce2d1d498\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2"
Nov 03 20:03:50.353837 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:50.353811    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slj5q\" (UniqueName: \"kubernetes.io/projected/9d9d77f5-8819-447e-b0a0-c27ce2d1d498-kube-api-access-slj5q\") pod \"payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2\" (UID: \"9d9d77f5-8819-447e-b0a0-c27ce2d1d498\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2"
Nov 03 20:03:50.460939 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:50.460288    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d9d77f5-8819-447e-b0a0-c27ce2d1d498-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2\" (UID: \"9d9d77f5-8819-447e-b0a0-c27ce2d1d498\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2"
Nov 03 20:03:50.460939 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:50.460464    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slj5q\" (UniqueName: \"kubernetes.io/projected/9d9d77f5-8819-447e-b0a0-c27ce2d1d498-kube-api-access-slj5q\") pod \"payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2\" (UID: \"9d9d77f5-8819-447e-b0a0-c27ce2d1d498\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2"
Nov 03 20:03:50.470417 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:50.463087    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d9d77f5-8819-447e-b0a0-c27ce2d1d498-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2\" (UID: \"9d9d77f5-8819-447e-b0a0-c27ce2d1d498\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2"
Nov 03 20:03:50.729611 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:50.729474    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slj5q\" (UniqueName: \"kubernetes.io/projected/9d9d77f5-8819-447e-b0a0-c27ce2d1d498-kube-api-access-slj5q\") pod \"payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2\" (UID: \"9d9d77f5-8819-447e-b0a0-c27ce2d1d498\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2"
Nov 03 20:03:50.947028 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:50.946980    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2"
Nov 03 20:03:50.960230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:50.960085    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6"
Nov 03 20:03:51.128182 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:51.128146    5591 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 03 20:03:51.128848 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:51.128811    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f" podUID="7aff0e54-0f19-41b1-af95-70aee78fea74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 03 20:03:51.146950 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:51.146919    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr" event={"ID":"122f235f-b27f-4342-8f25-f004502a0291","Type":"ContainerStarted","Data":"0271911efb89677e8334b8fd2dbf53ed5491b9960c6af61e8149894e5c5a8d56"}
Nov 03 20:03:51.147084 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:51.147006    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr"
Nov 03 20:03:51.170959 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:51.170900    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr" podStartSLOduration=3.170879878 podStartE2EDuration="3.170879878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:51.167795575 +0000 UTC m=+1579.940088594" watchObservedRunningTime="2026-11-03 20:03:51.170879878 +0000 UTC m=+1579.943172897"
Nov 03 20:03:51.367829 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:51.367804    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2"]
Nov 03 20:03:51.370765 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:03:51.370735    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d9d77f5_8819_447e_b0a0_c27ce2d1d498.slice/crio-e170adf957110c32bf88c376395c8fac7913e4494ee447750731ba3cbabad5f9 WatchSource:0}: Error finding container e170adf957110c32bf88c376395c8fac7913e4494ee447750731ba3cbabad5f9: Status 404 returned error can't find the container with id e170adf957110c32bf88c376395c8fac7913e4494ee447750731ba3cbabad5f9
Nov 03 20:03:52.096713 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:52.096623    5591 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body=
Nov 03 20:03:52.096986 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:52.096957    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f" podUID="7aff0e54-0f19-41b1-af95-70aee78fea74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused"
Nov 03 20:03:52.155793 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:52.155751    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2" event={"ID":"9d9d77f5-8819-447e-b0a0-c27ce2d1d498","Type":"ContainerStarted","Data":"42fc421eb86b99bbda2dbb18c1f0eaf5633c92dd78fbe4ed59273596eaaf14c7"}
Nov 03 20:03:52.156240 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:52.155820    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2" event={"ID":"9d9d77f5-8819-447e-b0a0-c27ce2d1d498","Type":"ContainerStarted","Data":"e170adf957110c32bf88c376395c8fac7913e4494ee447750731ba3cbabad5f9"}
Nov 03 20:03:52.156240 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:52.155851    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2"
Nov 03 20:03:52.208232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:52.204684    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2" podStartSLOduration=2.20466205 podStartE2EDuration="2.20466205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:52.17424956 +0000 UTC m=+1580.946542578" watchObservedRunningTime="2026-11-03 20:03:52.20466205 +0000 UTC m=+1580.976955071"
Nov 03 20:03:52.864616 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:52.864583    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4"
Nov 03 20:03:52.864833 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:52.864665    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp"
Nov 03 20:03:53.098381 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:53.098353    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:03:53.098663 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:53.098643    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:03:53.099311 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:53.099292    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:03:54.977712 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:54.977680    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:03:58.727044 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:03:58.727005    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:03:59.655810 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:59.655773    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6"]
Nov 03 20:03:59.661971 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:59.661952    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6"
Nov 03 20:03:59.668396 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:59.668361    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-bcfde8-serving-cert\""
Nov 03 20:03:59.674684 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:59.673483    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6"]
Nov 03 20:03:59.741771 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:59.741723    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6\" (UID: \"762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6"
Nov 03 20:03:59.742222 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:59.741943    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6\" (UID: \"762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6"
Nov 03 20:03:59.742222 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:59.742077    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdzph\" (UniqueName: \"kubernetes.io/projected/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-kube-api-access-hdzph\") pod \"maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6\" (UID: \"762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6"
Nov 03 20:03:59.742222 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:59.742156    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6\" (UID: \"762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6"
Nov 03 20:03:59.860617 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:59.853102    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6\" (UID: \"762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6"
Nov 03 20:03:59.861238 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:59.861180    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6\" (UID: \"762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6"
Nov 03 20:03:59.862442 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:03:59.862408    5591 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-serving-cert: secret "maas-api-e2e-shared-b-w3-bcfde8-serving-cert" not found
Nov 03 20:03:59.862587 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:03:59.862517    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-maas-api-tls podName:762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2 nodeName:}" failed. No retries permitted until 2026-11-03 20:04:00.362492162 +0000 UTC m=+1589.134785201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-maas-api-tls") pod "maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6" (UID: "762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2") : secret "maas-api-e2e-shared-b-w3-bcfde8-serving-cert" not found
Nov 03 20:03:59.862714 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:59.862689    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdzph\" (UniqueName: \"kubernetes.io/projected/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-kube-api-access-hdzph\") pod \"maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6\" (UID: \"762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6"
Nov 03 20:03:59.862851 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:59.862835    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6\" (UID: \"762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6"
Nov 03 20:03:59.863011 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:59.855014    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6\" (UID: \"762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6"
Nov 03 20:03:59.885780 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:59.871419    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6\" (UID: \"762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6"
Nov 03 20:03:59.893482 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:03:59.893442    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdzph\" (UniqueName: \"kubernetes.io/projected/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-kube-api-access-hdzph\") pod \"maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6\" (UID: \"762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6"
Nov 03 20:04:00.398234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:00.396437    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6\" (UID: \"762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6"
Nov 03 20:04:00.398234 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:04:00.396806    5591 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-serving-cert: secret "maas-api-e2e-shared-b-w3-bcfde8-serving-cert" not found
Nov 03 20:04:00.398234 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:04:00.397065    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-maas-api-tls podName:762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2 nodeName:}" failed. No retries permitted until 2026-11-03 20:04:01.396954672 +0000 UTC m=+1590.169247681 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-maas-api-tls") pod "maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6" (UID: "762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2") : secret "maas-api-e2e-shared-b-w3-bcfde8-serving-cert" not found
Nov 03 20:04:01.150655 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:01.150627    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc"
Nov 03 20:04:01.436412 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:01.431688    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6\" (UID: \"762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6"
Nov 03 20:04:01.455317 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:01.455236    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6\" (UID: \"762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6"
Nov 03 20:04:01.466561 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:01.466526    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq"]
Nov 03 20:04:01.472897 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:01.472867    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq"]
Nov 03 20:04:01.473074 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:01.472997    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq"
Nov 03 20:04:01.481268 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:01.479908    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6"
Nov 03 20:04:01.481268 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:01.480394    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-43c674-serving-cert\""
Nov 03 20:04:01.688938 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:01.679259    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22691b0c-82ca-4d24-85e6-fc861fcb73c0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq\" (UID: \"22691b0c-82ca-4d24-85e6-fc861fcb73c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq"
Nov 03 20:04:01.688938 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:01.679385    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bqvv\" (UniqueName: \"kubernetes.io/projected/22691b0c-82ca-4d24-85e6-fc861fcb73c0-kube-api-access-8bqvv\") pod \"maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq\" (UID: \"22691b0c-82ca-4d24-85e6-fc861fcb73c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq"
Nov 03 20:04:01.688938 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:01.679544    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22691b0c-82ca-4d24-85e6-fc861fcb73c0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq\" (UID: \"22691b0c-82ca-4d24-85e6-fc861fcb73c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq"
Nov 03 20:04:01.701250 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:01.701016    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22691b0c-82ca-4d24-85e6-fc861fcb73c0-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq\" (UID: \"22691b0c-82ca-4d24-85e6-fc861fcb73c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq"
Nov 03 20:04:01.802704 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:01.802667    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22691b0c-82ca-4d24-85e6-fc861fcb73c0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq\" (UID: \"22691b0c-82ca-4d24-85e6-fc861fcb73c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq"
Nov 03 20:04:01.802884 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:01.802763    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bqvv\" (UniqueName: \"kubernetes.io/projected/22691b0c-82ca-4d24-85e6-fc861fcb73c0-kube-api-access-8bqvv\") pod \"maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq\" (UID: \"22691b0c-82ca-4d24-85e6-fc861fcb73c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq"
Nov 03 20:04:01.802884 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:01.802836    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22691b0c-82ca-4d24-85e6-fc861fcb73c0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq\" (UID: \"22691b0c-82ca-4d24-85e6-fc861fcb73c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq"
Nov 03 20:04:01.802995 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:01.802979    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22691b0c-82ca-4d24-85e6-fc861fcb73c0-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq\" (UID: \"22691b0c-82ca-4d24-85e6-fc861fcb73c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq"
Nov 03 20:04:01.803539 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:01.803509    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22691b0c-82ca-4d24-85e6-fc861fcb73c0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq\" (UID: \"22691b0c-82ca-4d24-85e6-fc861fcb73c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq"
Nov 03 20:04:01.805533 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:01.805507    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22691b0c-82ca-4d24-85e6-fc861fcb73c0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq\" (UID: \"22691b0c-82ca-4d24-85e6-fc861fcb73c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq"
Nov 03 20:04:01.805643 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:01.805593    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22691b0c-82ca-4d24-85e6-fc861fcb73c0-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq\" (UID: \"22691b0c-82ca-4d24-85e6-fc861fcb73c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq"
Nov 03 20:04:01.816232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:01.816185    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bqvv\" (UniqueName: \"kubernetes.io/projected/22691b0c-82ca-4d24-85e6-fc861fcb73c0-kube-api-access-8bqvv\") pod \"maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq\" (UID: \"22691b0c-82ca-4d24-85e6-fc861fcb73c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq"
Nov 03 20:04:01.894987 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:01.894961    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6"]
Nov 03 20:04:01.896114 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:04:01.896091    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod762e5146_9a8e_49d6_b2a8_d3c9e46cfcd2.slice/crio-fe38fb0572686e0f36d1f85ebdedba9e8ea873fa8c9175bf5b638b3197972a90 WatchSource:0}: Error finding container fe38fb0572686e0f36d1f85ebdedba9e8ea873fa8c9175bf5b638b3197972a90: Status 404 returned error can't find the container with id fe38fb0572686e0f36d1f85ebdedba9e8ea873fa8c9175bf5b638b3197972a90
Nov 03 20:04:02.097273 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:02.097191    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq"
Nov 03 20:04:02.467266 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:02.467231    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq"]
Nov 03 20:04:02.470722 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:04:02.469326    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22691b0c_82ca_4d24_85e6_fc861fcb73c0.slice/crio-650fea909e0e9e2de2d0134791ee81285902cfff4098178680bec7aa3034fbc0 WatchSource:0}: Error finding container 650fea909e0e9e2de2d0134791ee81285902cfff4098178680bec7aa3034fbc0: Status 404 returned error can't find the container with id 650fea909e0e9e2de2d0134791ee81285902cfff4098178680bec7aa3034fbc0
Nov 03 20:04:02.488118 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:02.488075    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6" event={"ID":"762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2","Type":"ContainerStarted","Data":"fe38fb0572686e0f36d1f85ebdedba9e8ea873fa8c9175bf5b638b3197972a90"}
Nov 03 20:04:03.166739 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:03.166711    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2"
Nov 03 20:04:03.166915 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:03.166776    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr"
Nov 03 20:04:03.497978 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:03.497936    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq" event={"ID":"22691b0c-82ca-4d24-85e6-fc861fcb73c0","Type":"ContainerStarted","Data":"94fc9a9e588fa50991d89ce93e91e187ec4681ebc1b61ed279c3f9e9df668dd5"}
Nov 03 20:04:03.498576 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:03.498009    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq" event={"ID":"22691b0c-82ca-4d24-85e6-fc861fcb73c0","Type":"ContainerStarted","Data":"650fea909e0e9e2de2d0134791ee81285902cfff4098178680bec7aa3034fbc0"}
Nov 03 20:04:03.498576 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:03.498117    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq"
Nov 03 20:04:03.500240 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:03.500149    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6" event={"ID":"762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2","Type":"ContainerStarted","Data":"f44858d5f24ac8656eb764d913bb3adfba01ce13dfaecc932fa4bbc9b1dddd49"}
Nov 03 20:04:03.500398 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:03.500316    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6"
Nov 03 20:04:03.523652 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:03.523605    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq" podStartSLOduration=2.004551521 podStartE2EDuration="2.523591697s" totalImagesPullingTime="519.040176ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:03.519675637 +0000 UTC m=+1592.291968655" watchObservedRunningTime="2026-11-03 20:04:03.523591697 +0000 UTC m=+1592.295884714"
Nov 03 20:04:03.541114 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:03.541064    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6" podStartSLOduration=4.020306961 podStartE2EDuration="4.541048123s" totalImagesPullingTime="520.741162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:03.540007176 +0000 UTC m=+1592.312300196" watchObservedRunningTime="2026-11-03 20:04:03.541048123 +0000 UTC m=+1592.313341181"
Nov 03 20:04:09.519545 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:09.519515    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq"
Nov 03 20:04:09.520029 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:09.520008    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6"
Nov 03 20:04:11.233937 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.233903    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"]
Nov 03 20:04:11.241025 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.240994    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.270244 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.260812    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"]
Nov 03 20:04:11.329411 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.329381    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-bfc240-openshift-default-dockercfg-dxqxq\""
Nov 03 20:04:11.348084 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.348043    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-token\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.348278 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.348089    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d6b486-1932-4aec-ab87-770bb3f4db38-istiod-ca-cert\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.348278 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.348174    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-credential-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.348278 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.348265    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-data\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.348412 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.348312    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9nv9\" (UniqueName: \"kubernetes.io/projected/67d6b486-1932-4aec-ab87-770bb3f4db38-kube-api-access-n9nv9\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.348517 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.348492    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-envoy\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.348556 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.348541    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-workload-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.348635 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.348621    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-workload-certs\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.348687 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.348668    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-podinfo\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.450197 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.450147    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-token\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.450387 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.450220    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d6b486-1932-4aec-ab87-770bb3f4db38-istiod-ca-cert\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.450387 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.450254    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-credential-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.450387 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.450315    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-data\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.450387 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.450373    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9nv9\" (UniqueName: \"kubernetes.io/projected/67d6b486-1932-4aec-ab87-770bb3f4db38-kube-api-access-n9nv9\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.450627 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.450484    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-envoy\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.450627 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.450511    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-workload-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.450627 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.450561    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-workload-certs\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.450627 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.450593    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-podinfo\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.450965 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.450935    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-data\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.451603 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.451570    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-workload-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.451740 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.451602    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-workload-certs\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.451901 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.451798    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d6b486-1932-4aec-ab87-770bb3f4db38-istiod-ca-cert\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.451901 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.451800    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-credential-socket\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.453447 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.453427    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-podinfo\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.453605 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.453576    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-envoy\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.463849 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.463822    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9nv9\" (UniqueName: \"kubernetes.io/projected/67d6b486-1932-4aec-ab87-770bb3f4db38-kube-api-access-n9nv9\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.464667 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.464632    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-token\") pod \"e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") " pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.597832 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.597527    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:11.961326 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.961241    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"]
Nov 03 20:04:11.967468 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:04:11.967416    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d6b486_1932_4aec_ab87_770bb3f4db38.slice/crio-dff926f37d46da6a44bf2a264aeb01d6075788cdf95fb7fefb5d3b712bf2187b WatchSource:0}: Error finding container dff926f37d46da6a44bf2a264aeb01d6075788cdf95fb7fefb5d3b712bf2187b: Status 404 returned error can't find the container with id dff926f37d46da6a44bf2a264aeb01d6075788cdf95fb7fefb5d3b712bf2187b
Nov 03 20:04:11.971265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.971220    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:04:11.971395 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.971321    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:04:11.971395 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:11.971368    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:04:12.673753 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:12.673708    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz" event={"ID":"67d6b486-1932-4aec-ab87-770bb3f4db38","Type":"ContainerStarted","Data":"f93c7189b463f1c52bb4b99c31dbeb4edcc45522723255d8d4ce2833085a5d90"}
Nov 03 20:04:12.674138 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:12.673771    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz" event={"ID":"67d6b486-1932-4aec-ab87-770bb3f4db38","Type":"ContainerStarted","Data":"dff926f37d46da6a44bf2a264aeb01d6075788cdf95fb7fefb5d3b712bf2187b"}
Nov 03 20:04:12.700846 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:12.700793    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz" podStartSLOduration=1.700772004 podStartE2EDuration="1.700772004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:12.698302818 +0000 UTC m=+1601.470595831" watchObservedRunningTime="2026-11-03 20:04:12.700772004 +0000 UTC m=+1601.473065021"
Nov 03 20:04:13.600230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:13.599842    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:13.607433 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:13.607393    5591 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 03 20:04:13.607624 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:13.607473    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz" podUID="67d6b486-1932-4aec-ab87-770bb3f4db38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 03 20:04:14.605228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:14.605002    5591 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 03 20:04:14.605228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:14.605073    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz" podUID="67d6b486-1932-4aec-ab87-770bb3f4db38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 03 20:04:15.604307 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:15.603477    5591 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 03 20:04:15.604307 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:15.603548    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz" podUID="67d6b486-1932-4aec-ab87-770bb3f4db38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 03 20:04:16.610296 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:16.608782    5591 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 03 20:04:16.610296 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:16.608854    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz" podUID="67d6b486-1932-4aec-ab87-770bb3f4db38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 03 20:04:17.614274 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:17.614193    5591 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 03 20:04:17.614774 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:17.614319    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz" podUID="67d6b486-1932-4aec-ab87-770bb3f4db38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 03 20:04:18.605391 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:18.605353    5591 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 03 20:04:18.605622 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:18.605418    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz" podUID="67d6b486-1932-4aec-ab87-770bb3f4db38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 03 20:04:19.590230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:19.583430    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:04:19.712969 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:19.703971    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:19.712969 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:19.711470    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:19.714437 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:19.714411    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:04:28.736543 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:04:28.736501    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:04:35.732251 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:04:35.732197    5591 scope.go:122] "RemoveContainer" containerID="6a575b26f292570d4f7d08bcbac3c47c245a49ce6835d60ed76f5bf50e1297e5"
Nov 03 20:04:58.745109 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:04:58.745071    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:05:02.048004 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:02.047966    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:05:11.883167 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:11.883126    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"]
Nov 03 20:05:11.889378 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:11.889354    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"
Nov 03 20:05:11.896738 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:11.896712    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bcfde8\"/\"kube-root-ca.crt\""
Nov 03 20:05:11.897509 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:11.897488    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bcfde8\"/\"openshift-service-ca.crt\""
Nov 03 20:05:11.897635 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:11.897504    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bcfde8\"/\"auth-test-model-bcfde8-kserve-self-signed-certs\""
Nov 03 20:05:11.897698 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:11.897652    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bcfde8\"/\"default-dockercfg-jwcxg\""
Nov 03 20:05:11.900602 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:11.900580    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"]
Nov 03 20:05:12.021682 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:12.021650    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-model-cache\") pod \"auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2\" (UID: \"9f7d28df-3147-416e-aebb-d755624f00e4\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"
Nov 03 20:05:12.021850 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:12.021729    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zjnt\" (UniqueName: \"kubernetes.io/projected/9f7d28df-3147-416e-aebb-d755624f00e4-kube-api-access-5zjnt\") pod \"auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2\" (UID: \"9f7d28df-3147-416e-aebb-d755624f00e4\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"
Nov 03 20:05:12.021850 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:12.021806    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-tmp-dir\") pod \"auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2\" (UID: \"9f7d28df-3147-416e-aebb-d755624f00e4\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"
Nov 03 20:05:12.021927 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:12.021856    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-dshm\") pod \"auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2\" (UID: \"9f7d28df-3147-416e-aebb-d755624f00e4\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"
Nov 03 20:05:12.022002 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:12.021982    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-home\") pod \"auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2\" (UID: \"9f7d28df-3147-416e-aebb-d755624f00e4\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"
Nov 03 20:05:12.022134 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:12.022121    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f7d28df-3147-416e-aebb-d755624f00e4-tls-certs\") pod \"auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2\" (UID: \"9f7d28df-3147-416e-aebb-d755624f00e4\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"
Nov 03 20:05:12.135464 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:12.135370    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f7d28df-3147-416e-aebb-d755624f00e4-tls-certs\") pod \"auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2\" (UID: \"9f7d28df-3147-416e-aebb-d755624f00e4\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"
Nov 03 20:05:12.174230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:12.162717    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-model-cache\") pod \"auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2\" (UID: \"9f7d28df-3147-416e-aebb-d755624f00e4\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"
Nov 03 20:05:12.174230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:12.163406    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zjnt\" (UniqueName: \"kubernetes.io/projected/9f7d28df-3147-416e-aebb-d755624f00e4-kube-api-access-5zjnt\") pod \"auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2\" (UID: \"9f7d28df-3147-416e-aebb-d755624f00e4\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"
Nov 03 20:05:12.174230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:12.163449    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-tmp-dir\") pod \"auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2\" (UID: \"9f7d28df-3147-416e-aebb-d755624f00e4\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"
Nov 03 20:05:12.174230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:12.163517    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-dshm\") pod \"auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2\" (UID: \"9f7d28df-3147-416e-aebb-d755624f00e4\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"
Nov 03 20:05:12.174230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:12.163166    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-model-cache\") pod \"auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2\" (UID: \"9f7d28df-3147-416e-aebb-d755624f00e4\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"
Nov 03 20:05:12.174230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:12.164277    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-tmp-dir\") pod \"auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2\" (UID: \"9f7d28df-3147-416e-aebb-d755624f00e4\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"
Nov 03 20:05:12.174230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:12.164374    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-home\") pod \"auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2\" (UID: \"9f7d28df-3147-416e-aebb-d755624f00e4\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"
Nov 03 20:05:12.174230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:12.164776    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-home\") pod \"auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2\" (UID: \"9f7d28df-3147-416e-aebb-d755624f00e4\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"
Nov 03 20:05:12.174230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:12.165808    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f7d28df-3147-416e-aebb-d755624f00e4-tls-certs\") pod \"auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2\" (UID: \"9f7d28df-3147-416e-aebb-d755624f00e4\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"
Nov 03 20:05:12.210034 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:12.209015    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-dshm\") pod \"auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2\" (UID: \"9f7d28df-3147-416e-aebb-d755624f00e4\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"
Nov 03 20:05:12.246359 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:12.246290    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zjnt\" (UniqueName: \"kubernetes.io/projected/9f7d28df-3147-416e-aebb-d755624f00e4-kube-api-access-5zjnt\") pod \"auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2\" (UID: \"9f7d28df-3147-416e-aebb-d755624f00e4\") " pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"
Nov 03 20:05:12.505485 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:12.505343    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"
Nov 03 20:05:12.889256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:12.889224    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"]
Nov 03 20:05:12.889959 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:05:12.889935    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f7d28df_3147_416e_aebb_d755624f00e4.slice/crio-47a486c4dbe42d56b14ed882527d03438e93241b74abe4f4be468d8fe1b804a3 WatchSource:0}: Error finding container 47a486c4dbe42d56b14ed882527d03438e93241b74abe4f4be468d8fe1b804a3: Status 404 returned error can't find the container with id 47a486c4dbe42d56b14ed882527d03438e93241b74abe4f4be468d8fe1b804a3
Nov 03 20:05:13.662822 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:13.662788    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2" event={"ID":"9f7d28df-3147-416e-aebb-d755624f00e4","Type":"ContainerStarted","Data":"e23a86a66d007908f50798f95b0f0b95b42d1115b870bbcb74d5e083a80ca7ac"}
Nov 03 20:05:13.663009 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:13.662833    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2" event={"ID":"9f7d28df-3147-416e-aebb-d755624f00e4","Type":"ContainerStarted","Data":"47a486c4dbe42d56b14ed882527d03438e93241b74abe4f4be468d8fe1b804a3"}
Nov 03 20:05:13.688506 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:13.688450    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2" podStartSLOduration=2.688436933 podStartE2EDuration="2.688436933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:13.684506147 +0000 UTC m=+1662.456799167" watchObservedRunningTime="2026-11-03 20:05:13.688436933 +0000 UTC m=+1662.460729989"
Nov 03 20:05:14.505736 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:14.505701    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"
Nov 03 20:05:22.505621 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:22.505587    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"
Nov 03 20:05:22.519474 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:22.519446    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"
Nov 03 20:05:22.782548 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:22.782520    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"
Nov 03 20:05:28.724339 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:05:28.724302    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:05:48.499760 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:48.499722    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"]
Nov 03 20:05:48.529694 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:48.529651    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"]
Nov 03 20:05:48.529887 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:48.529781    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"
Nov 03 20:05:48.539190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:48.539133    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-a44263\"/\"default-dockercfg-ljhf9\""
Nov 03 20:05:48.539314 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:48.539280    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-a44263\"/\"noten-model-a44263-kserve-self-signed-certs\""
Nov 03 20:05:48.540281 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:48.540259    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-a44263\"/\"kube-root-ca.crt\""
Nov 03 20:05:48.540437 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:48.540341    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-a44263\"/\"openshift-service-ca.crt\""
Nov 03 20:05:48.617968 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:48.617927    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d948091-5653-4673-90af-3a9c59918e8a-tls-certs\") pod \"noten-model-a44263-kserve-5b4fc74fdf-6xcsb\" (UID: \"6d948091-5653-4673-90af-3a9c59918e8a\") " pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"
Nov 03 20:05:48.618223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:48.618098    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-dshm\") pod \"noten-model-a44263-kserve-5b4fc74fdf-6xcsb\" (UID: \"6d948091-5653-4673-90af-3a9c59918e8a\") " pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"
Nov 03 20:05:48.618323 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:48.618226    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-model-cache\") pod \"noten-model-a44263-kserve-5b4fc74fdf-6xcsb\" (UID: \"6d948091-5653-4673-90af-3a9c59918e8a\") " pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"
Nov 03 20:05:48.618381 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:48.618352    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-home\") pod \"noten-model-a44263-kserve-5b4fc74fdf-6xcsb\" (UID: \"6d948091-5653-4673-90af-3a9c59918e8a\") " pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"
Nov 03 20:05:48.618430 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:48.618392    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lznw6\" (UniqueName: \"kubernetes.io/projected/6d948091-5653-4673-90af-3a9c59918e8a-kube-api-access-lznw6\") pod \"noten-model-a44263-kserve-5b4fc74fdf-6xcsb\" (UID: \"6d948091-5653-4673-90af-3a9c59918e8a\") " pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"
Nov 03 20:05:48.618582 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:48.618557    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-tmp-dir\") pod \"noten-model-a44263-kserve-5b4fc74fdf-6xcsb\" (UID: \"6d948091-5653-4673-90af-3a9c59918e8a\") " pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"
Nov 03 20:05:48.729929 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:48.729886    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-tmp-dir\") pod \"noten-model-a44263-kserve-5b4fc74fdf-6xcsb\" (UID: \"6d948091-5653-4673-90af-3a9c59918e8a\") " pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"
Nov 03 20:05:48.730505 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:48.730467    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d948091-5653-4673-90af-3a9c59918e8a-tls-certs\") pod \"noten-model-a44263-kserve-5b4fc74fdf-6xcsb\" (UID: \"6d948091-5653-4673-90af-3a9c59918e8a\") " pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"
Nov 03 20:05:48.755335 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:48.731569    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-tmp-dir\") pod \"noten-model-a44263-kserve-5b4fc74fdf-6xcsb\" (UID: \"6d948091-5653-4673-90af-3a9c59918e8a\") " pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"
Nov 03 20:05:48.755335 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:48.742901    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d948091-5653-4673-90af-3a9c59918e8a-tls-certs\") pod \"noten-model-a44263-kserve-5b4fc74fdf-6xcsb\" (UID: \"6d948091-5653-4673-90af-3a9c59918e8a\") " pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"
Nov 03 20:05:48.755792 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:48.755762    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-dshm\") pod \"noten-model-a44263-kserve-5b4fc74fdf-6xcsb\" (UID: \"6d948091-5653-4673-90af-3a9c59918e8a\") " pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"
Nov 03 20:05:48.756079 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:48.756055    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-model-cache\") pod \"noten-model-a44263-kserve-5b4fc74fdf-6xcsb\" (UID: \"6d948091-5653-4673-90af-3a9c59918e8a\") " pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"
Nov 03 20:05:48.756441 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:48.756407    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-home\") pod \"noten-model-a44263-kserve-5b4fc74fdf-6xcsb\" (UID: \"6d948091-5653-4673-90af-3a9c59918e8a\") " pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"
Nov 03 20:05:48.756743 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:48.756613    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lznw6\" (UniqueName: \"kubernetes.io/projected/6d948091-5653-4673-90af-3a9c59918e8a-kube-api-access-lznw6\") pod \"noten-model-a44263-kserve-5b4fc74fdf-6xcsb\" (UID: \"6d948091-5653-4673-90af-3a9c59918e8a\") " pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"
Nov 03 20:05:48.760236 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:48.759558    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-home\") pod \"noten-model-a44263-kserve-5b4fc74fdf-6xcsb\" (UID: \"6d948091-5653-4673-90af-3a9c59918e8a\") " pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"
Nov 03 20:05:48.773255 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:48.766361    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-model-cache\") pod \"noten-model-a44263-kserve-5b4fc74fdf-6xcsb\" (UID: \"6d948091-5653-4673-90af-3a9c59918e8a\") " pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"
Nov 03 20:05:48.773255 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:48.766965    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-dshm\") pod \"noten-model-a44263-kserve-5b4fc74fdf-6xcsb\" (UID: \"6d948091-5653-4673-90af-3a9c59918e8a\") " pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"
Nov 03 20:05:48.816233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:48.797458    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lznw6\" (UniqueName: \"kubernetes.io/projected/6d948091-5653-4673-90af-3a9c59918e8a-kube-api-access-lznw6\") pod \"noten-model-a44263-kserve-5b4fc74fdf-6xcsb\" (UID: \"6d948091-5653-4673-90af-3a9c59918e8a\") " pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"
Nov 03 20:05:48.857250 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:48.856525    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"
Nov 03 20:05:49.187974 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:49.187943    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"]
Nov 03 20:05:50.196239 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:50.196192    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb" event={"ID":"6d948091-5653-4673-90af-3a9c59918e8a","Type":"ContainerStarted","Data":"34d78a278284814732969988a1c4c38789e553b2b497e15297f30a4a6ca6f5a3"}
Nov 03 20:05:50.196753 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:50.196251    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb" event={"ID":"6d948091-5653-4673-90af-3a9c59918e8a","Type":"ContainerStarted","Data":"2b34355b5b7804bbc86c9899b3ebb1f86b34b3d4f733c28edd19bf2998082f46"}
Nov 03 20:05:50.226950 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:50.226899    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb" podStartSLOduration=2.226885579 podStartE2EDuration="2.226885579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:50.223843775 +0000 UTC m=+1698.996136795" watchObservedRunningTime="2026-11-03 20:05:50.226885579 +0000 UTC m=+1698.999178597"
Nov 03 20:05:50.858598 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:50.858567    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"
Nov 03 20:05:52.538184 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:52.538108    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:05:56.303965 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:56.303929    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"]
Nov 03 20:05:56.304562 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:56.304121    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q" podUID="68b82f0c-e037-4366-891b-1ee898b2cba2" containerName="istio-proxy" containerID="cri-o://64533ad18d753c4cf532a3950904c061dff824d13699ca5de5480a4e40535b11" gracePeriod=30
Nov 03 20:05:58.727638 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:05:58.727603    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:05:58.860519 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:58.860485    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"
Nov 03 20:05:58.874097 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:58.874067    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"
Nov 03 20:05:59.275729 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:59.275703    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"
Nov 03 20:05:59.896801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:05:59.896757    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"]
Nov 03 20:06:00.429253 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:06:00.429219    5591 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-self-signed-certs: secret "noten-model-a44263-kserve-self-signed-certs" not found
Nov 03 20:06:00.429592 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:06:00.429577    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d948091-5653-4673-90af-3a9c59918e8a-tls-certs podName:6d948091-5653-4673-90af-3a9c59918e8a nodeName:}" failed. No retries permitted until 2026-11-03 20:06:00.929535677 +0000 UTC m=+1709.701828690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6d948091-5653-4673-90af-3a9c59918e8a-tls-certs") pod "noten-model-a44263-kserve-5b4fc74fdf-6xcsb" (UID: "6d948091-5653-4673-90af-3a9c59918e8a") : secret "noten-model-a44263-kserve-self-signed-certs" not found
Nov 03 20:06:00.808345 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:00.808308    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"]
Nov 03 20:06:00.808636 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:00.808596    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989" podUID="1faa21e4-5bf7-418d-adc8-3fc5bf955010" containerName="istio-proxy" containerID="cri-o://90f25ed1b925a5da58add530a5b9b74af4d57ab007e8898b05ce218141167469" gracePeriod=30
Nov 03 20:06:00.954246 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:06:00.946014    5591 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-self-signed-certs: secret "noten-model-a44263-kserve-self-signed-certs" not found
Nov 03 20:06:00.955479 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:06:00.955453    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d948091-5653-4673-90af-3a9c59918e8a-tls-certs podName:6d948091-5653-4673-90af-3a9c59918e8a nodeName:}" failed. No retries permitted until 2026-11-03 20:06:01.955424296 +0000 UTC m=+1710.727717304 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6d948091-5653-4673-90af-3a9c59918e8a-tls-certs") pod "noten-model-a44263-kserve-5b4fc74fdf-6xcsb" (UID: "6d948091-5653-4673-90af-3a9c59918e8a") : secret "noten-model-a44263-kserve-self-signed-certs" not found
Nov 03 20:06:01.725175 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.725144    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 20:06:01.831690 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.831659    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-data\") pod \"68b82f0c-e037-4366-891b-1ee898b2cba2\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") "
Nov 03 20:06:01.831888 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.831711    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-workload-certs\") pod \"68b82f0c-e037-4366-891b-1ee898b2cba2\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") "
Nov 03 20:06:01.831888 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.831833    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-credential-socket\") pod \"68b82f0c-e037-4366-891b-1ee898b2cba2\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") "
Nov 03 20:06:01.831888 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.831876    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-podinfo\") pod \"68b82f0c-e037-4366-891b-1ee898b2cba2\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") "
Nov 03 20:06:01.832018 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.831975    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-envoy\") pod \"68b82f0c-e037-4366-891b-1ee898b2cba2\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") "
Nov 03 20:06:01.832018 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.831982    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-workload-certs" pod "68b82f0c-e037-4366-891b-1ee898b2cba2" (UID: "68b82f0c-e037-4366-891b-1ee898b2cba2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:01.832097 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.831993    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-data" pod "68b82f0c-e037-4366-891b-1ee898b2cba2" (UID: "68b82f0c-e037-4366-891b-1ee898b2cba2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:01.832097 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.832046    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68b82f0c-e037-4366-891b-1ee898b2cba2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b82f0c-e037-4366-891b-1ee898b2cba2-istiod-ca-cert\") pod \"68b82f0c-e037-4366-891b-1ee898b2cba2\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") "
Nov 03 20:06:01.832097 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.832070    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-credential-socket" pod "68b82f0c-e037-4366-891b-1ee898b2cba2" (UID: "68b82f0c-e037-4366-891b-1ee898b2cba2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:01.832097 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.832088    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-workload-socket\") pod \"68b82f0c-e037-4366-891b-1ee898b2cba2\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") "
Nov 03 20:06:01.832305 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.832120    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b82f0c-e037-4366-891b-1ee898b2cba2-kube-api-access-s7r8g\" (UniqueName: \"kubernetes.io/projected/68b82f0c-e037-4366-891b-1ee898b2cba2-kube-api-access-s7r8g\") pod \"68b82f0c-e037-4366-891b-1ee898b2cba2\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") "
Nov 03 20:06:01.832305 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.832152    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-token\" (UniqueName: \"kubernetes.io/projected/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-token\") pod \"68b82f0c-e037-4366-891b-1ee898b2cba2\" (UID: \"68b82f0c-e037-4366-891b-1ee898b2cba2\") "
Nov 03 20:06:01.832423 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.832337    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-workload-socket" pod "68b82f0c-e037-4366-891b-1ee898b2cba2" (UID: "68b82f0c-e037-4366-891b-1ee898b2cba2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:01.832423 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.832389    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68b82f0c-e037-4366-891b-1ee898b2cba2-istiod-ca-cert" pod "68b82f0c-e037-4366-891b-1ee898b2cba2" (UID: "68b82f0c-e037-4366-891b-1ee898b2cba2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:01.833159 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.833128    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:01.833159 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.833159    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b82f0c-e037-4366-891b-1ee898b2cba2-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:01.833411 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.833175    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:01.833411 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.833190    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:01.833411 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.833224    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:01.834534 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.834501    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-envoy" pod "68b82f0c-e037-4366-891b-1ee898b2cba2" (UID: "68b82f0c-e037-4366-891b-1ee898b2cba2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:01.834675 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.834659    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-podinfo" pod "68b82f0c-e037-4366-891b-1ee898b2cba2" (UID: "68b82f0c-e037-4366-891b-1ee898b2cba2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:06:01.835012 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.834997    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b82f0c-e037-4366-891b-1ee898b2cba2-kube-api-access-s7r8g" pod "68b82f0c-e037-4366-891b-1ee898b2cba2" (UID: "68b82f0c-e037-4366-891b-1ee898b2cba2"). InnerVolumeSpecName "kube-api-access-s7r8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:01.835482 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.835455    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-token" pod "68b82f0c-e037-4366-891b-1ee898b2cba2" (UID: "68b82f0c-e037-4366-891b-1ee898b2cba2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:01.934534 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.934497    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:01.934534 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.934528    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7r8g\" (UniqueName: \"kubernetes.io/projected/68b82f0c-e037-4366-891b-1ee898b2cba2-kube-api-access-s7r8g\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:01.934534 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.934542    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:01.934789 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:01.934556    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b82f0c-e037-4366-891b-1ee898b2cba2-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:02.036014 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:06:02.035983    5591 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-self-signed-certs: secret "noten-model-a44263-kserve-self-signed-certs" not found
Nov 03 20:06:02.036467 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:06:02.036069    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d948091-5653-4673-90af-3a9c59918e8a-tls-certs podName:6d948091-5653-4673-90af-3a9c59918e8a nodeName:}" failed. No retries permitted until 2026-11-03 20:06:04.036050023 +0000 UTC m=+1712.808343041 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6d948091-5653-4673-90af-3a9c59918e8a-tls-certs") pod "noten-model-a44263-kserve-5b4fc74fdf-6xcsb" (UID: "6d948091-5653-4673-90af-3a9c59918e8a") : secret "noten-model-a44263-kserve-self-signed-certs" not found
Nov 03 20:06:02.360245 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:02.353146    5591 generic.go:356] "Generic (PLEG): container finished" podID="68b82f0c-e037-4366-891b-1ee898b2cba2" containerID="64533ad18d753c4cf532a3950904c061dff824d13699ca5de5480a4e40535b11" exitCode=0
Nov 03 20:06:02.360245 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:02.353329    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q" event={"ID":"68b82f0c-e037-4366-891b-1ee898b2cba2","Type":"ContainerDied","Data":"64533ad18d753c4cf532a3950904c061dff824d13699ca5de5480a4e40535b11"}
Nov 03 20:06:02.360245 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:02.353380    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q" event={"ID":"68b82f0c-e037-4366-891b-1ee898b2cba2","Type":"ContainerDied","Data":"e112b9e913494860fcce82bbe22b77e12b947106b9e4f9676e934205c2576d04"}
Nov 03 20:06:02.360245 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:02.353409    5591 scope.go:122] "RemoveContainer" containerID="64533ad18d753c4cf532a3950904c061dff824d13699ca5de5480a4e40535b11"
Nov 03 20:06:02.360245 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:02.353577    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"
Nov 03 20:06:02.372030 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:02.372002    5591 scope.go:122] "RemoveContainer" containerID="64533ad18d753c4cf532a3950904c061dff824d13699ca5de5480a4e40535b11"
Nov 03 20:06:02.372420 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:06:02.372401    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64533ad18d753c4cf532a3950904c061dff824d13699ca5de5480a4e40535b11\": container with ID starting with 64533ad18d753c4cf532a3950904c061dff824d13699ca5de5480a4e40535b11 not found: ID does not exist" containerID="64533ad18d753c4cf532a3950904c061dff824d13699ca5de5480a4e40535b11"
Nov 03 20:06:02.372489 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:02.372429    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64533ad18d753c4cf532a3950904c061dff824d13699ca5de5480a4e40535b11"} err="failed to get container status \"64533ad18d753c4cf532a3950904c061dff824d13699ca5de5480a4e40535b11\": rpc error: code = NotFound desc = could not find container \"64533ad18d753c4cf532a3950904c061dff824d13699ca5de5480a4e40535b11\": container with ID starting with 64533ad18d753c4cf532a3950904c061dff824d13699ca5de5480a4e40535b11 not found: ID does not exist"
Nov 03 20:06:02.383141 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:02.383114    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"]
Nov 03 20:06:02.389472 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:02.389444    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-t8m7q"]
Nov 03 20:06:03.352512 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:03.352481    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68b82f0c-e037-4366-891b-1ee898b2cba2" path="/var/lib/kubelet/pods/68b82f0c-e037-4366-891b-1ee898b2cba2/volumes"
Nov 03 20:06:04.080042 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:06:04.080004    5591 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-self-signed-certs: secret "noten-model-a44263-kserve-self-signed-certs" not found
Nov 03 20:06:04.080259 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:06:04.080093    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d948091-5653-4673-90af-3a9c59918e8a-tls-certs podName:6d948091-5653-4673-90af-3a9c59918e8a nodeName:}" failed. No retries permitted until 2026-11-03 20:06:08.080077125 +0000 UTC m=+1716.852370121 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6d948091-5653-4673-90af-3a9c59918e8a-tls-certs") pod "noten-model-a44263-kserve-5b4fc74fdf-6xcsb" (UID: "6d948091-5653-4673-90af-3a9c59918e8a") : secret "noten-model-a44263-kserve-self-signed-certs" not found
Nov 03 20:06:06.131881 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.131853    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 20:06:06.231330 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.229436    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istiod-ca-cert\") pod \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") "
Nov 03 20:06:06.231330 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.229493    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-podinfo\") pod \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") "
Nov 03 20:06:06.231330 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.229644    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1faa21e4-5bf7-418d-adc8-3fc5bf955010-kube-api-access-6tz2f\" (UniqueName: \"kubernetes.io/projected/1faa21e4-5bf7-418d-adc8-3fc5bf955010-kube-api-access-6tz2f\") pod \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") "
Nov 03 20:06:06.231330 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.229691    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-workload-socket\") pod \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") "
Nov 03 20:06:06.231330 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.229718    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-credential-socket\") pod \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") "
Nov 03 20:06:06.231330 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.229854    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-data\") pod \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") "
Nov 03 20:06:06.231330 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.229893    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-workload-certs\") pod \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") "
Nov 03 20:06:06.231330 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.230014    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-envoy\") pod \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") "
Nov 03 20:06:06.231330 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.230066    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-token\" (UniqueName: \"kubernetes.io/projected/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-token\") pod \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\" (UID: \"1faa21e4-5bf7-418d-adc8-3fc5bf955010\") "
Nov 03 20:06:06.233798 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.233765    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-workload-socket" pod "1faa21e4-5bf7-418d-adc8-3fc5bf955010" (UID: "1faa21e4-5bf7-418d-adc8-3fc5bf955010"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:06.262594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.262027    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-podinfo" pod "1faa21e4-5bf7-418d-adc8-3fc5bf955010" (UID: "1faa21e4-5bf7-418d-adc8-3fc5bf955010"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:06:06.262594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.262316    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-credential-socket" pod "1faa21e4-5bf7-418d-adc8-3fc5bf955010" (UID: "1faa21e4-5bf7-418d-adc8-3fc5bf955010"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:06.263315 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.263287    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-workload-certs" pod "1faa21e4-5bf7-418d-adc8-3fc5bf955010" (UID: "1faa21e4-5bf7-418d-adc8-3fc5bf955010"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:06.264331 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.264284    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-data" pod "1faa21e4-5bf7-418d-adc8-3fc5bf955010" (UID: "1faa21e4-5bf7-418d-adc8-3fc5bf955010"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:06.275682 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.275620    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istiod-ca-cert" pod "1faa21e4-5bf7-418d-adc8-3fc5bf955010" (UID: "1faa21e4-5bf7-418d-adc8-3fc5bf955010"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:06:06.276837 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.276786    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1faa21e4-5bf7-418d-adc8-3fc5bf955010-kube-api-access-6tz2f" pod "1faa21e4-5bf7-418d-adc8-3fc5bf955010" (UID: "1faa21e4-5bf7-418d-adc8-3fc5bf955010"). InnerVolumeSpecName "kube-api-access-6tz2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:06.285452 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.285394    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-envoy" pod "1faa21e4-5bf7-418d-adc8-3fc5bf955010" (UID: "1faa21e4-5bf7-418d-adc8-3fc5bf955010"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:06.328908 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.328850    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-token" pod "1faa21e4-5bf7-418d-adc8-3fc5bf955010" (UID: "1faa21e4-5bf7-418d-adc8-3fc5bf955010"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:06.332532 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.332363    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:06.332532 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.332395    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:06.332532 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.332412    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tz2f\" (UniqueName: \"kubernetes.io/projected/1faa21e4-5bf7-418d-adc8-3fc5bf955010-kube-api-access-6tz2f\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:06.332532 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.332427    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:06.332532 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.332440    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:06.332532 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.332452    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:06.332532 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.332465    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:06.332532 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.332477    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:06.332532 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.332492    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1faa21e4-5bf7-418d-adc8-3fc5bf955010-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:06.424658 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.424557    5591 generic.go:356] "Generic (PLEG): container finished" podID="1faa21e4-5bf7-418d-adc8-3fc5bf955010" containerID="90f25ed1b925a5da58add530a5b9b74af4d57ab007e8898b05ce218141167469" exitCode=0
Nov 03 20:06:06.424658 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.424608    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"
Nov 03 20:06:06.424658 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.424643    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989" event={"ID":"1faa21e4-5bf7-418d-adc8-3fc5bf955010","Type":"ContainerDied","Data":"90f25ed1b925a5da58add530a5b9b74af4d57ab007e8898b05ce218141167469"}
Nov 03 20:06:06.424927 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.424700    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989" event={"ID":"1faa21e4-5bf7-418d-adc8-3fc5bf955010","Type":"ContainerDied","Data":"b99c83f5b14b6c9d0d2d24d121db2491e08b97b625d27ebdb8228cc330c610a2"}
Nov 03 20:06:06.424927 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.424729    5591 scope.go:122] "RemoveContainer" containerID="90f25ed1b925a5da58add530a5b9b74af4d57ab007e8898b05ce218141167469"
Nov 03 20:06:06.436895 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.436875    5591 scope.go:122] "RemoveContainer" containerID="382ad4b105b08bb53ffbcbad01efa24da8a59243875b635ae171dba58a67a149"
Nov 03 20:06:06.447890 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.447869    5591 scope.go:122] "RemoveContainer" containerID="90f25ed1b925a5da58add530a5b9b74af4d57ab007e8898b05ce218141167469"
Nov 03 20:06:06.448280 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:06:06.448253    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90f25ed1b925a5da58add530a5b9b74af4d57ab007e8898b05ce218141167469\": container with ID starting with 90f25ed1b925a5da58add530a5b9b74af4d57ab007e8898b05ce218141167469 not found: ID does not exist" containerID="90f25ed1b925a5da58add530a5b9b74af4d57ab007e8898b05ce218141167469"
Nov 03 20:06:06.448368 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.448287    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90f25ed1b925a5da58add530a5b9b74af4d57ab007e8898b05ce218141167469"} err="failed to get container status \"90f25ed1b925a5da58add530a5b9b74af4d57ab007e8898b05ce218141167469\": rpc error: code = NotFound desc = could not find container \"90f25ed1b925a5da58add530a5b9b74af4d57ab007e8898b05ce218141167469\": container with ID starting with 90f25ed1b925a5da58add530a5b9b74af4d57ab007e8898b05ce218141167469 not found: ID does not exist"
Nov 03 20:06:06.448368 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.448305    5591 scope.go:122] "RemoveContainer" containerID="382ad4b105b08bb53ffbcbad01efa24da8a59243875b635ae171dba58a67a149"
Nov 03 20:06:06.448621 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:06:06.448603    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"382ad4b105b08bb53ffbcbad01efa24da8a59243875b635ae171dba58a67a149\": container with ID starting with 382ad4b105b08bb53ffbcbad01efa24da8a59243875b635ae171dba58a67a149 not found: ID does not exist" containerID="382ad4b105b08bb53ffbcbad01efa24da8a59243875b635ae171dba58a67a149"
Nov 03 20:06:06.448664 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.448630    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"382ad4b105b08bb53ffbcbad01efa24da8a59243875b635ae171dba58a67a149"} err="failed to get container status \"382ad4b105b08bb53ffbcbad01efa24da8a59243875b635ae171dba58a67a149\": rpc error: code = NotFound desc = could not find container \"382ad4b105b08bb53ffbcbad01efa24da8a59243875b635ae171dba58a67a149\": container with ID starting with 382ad4b105b08bb53ffbcbad01efa24da8a59243875b635ae171dba58a67a149 not found: ID does not exist"
Nov 03 20:06:06.451734 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.451710    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"]
Nov 03 20:06:06.457023 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:06.456998    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-zn989"]
Nov 03 20:06:07.351845 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:07.351812    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1faa21e4-5bf7-418d-adc8-3fc5bf955010" path="/var/lib/kubelet/pods/1faa21e4-5bf7-418d-adc8-3fc5bf955010/volumes"
Nov 03 20:06:08.162397 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:06:08.162368    5591 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-self-signed-certs: secret "noten-model-a44263-kserve-self-signed-certs" not found
Nov 03 20:06:08.162612 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:06:08.162448    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d948091-5653-4673-90af-3a9c59918e8a-tls-certs podName:6d948091-5653-4673-90af-3a9c59918e8a nodeName:}" failed. No retries permitted until 2026-11-03 20:06:16.162432697 +0000 UTC m=+1724.934725694 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6d948091-5653-4673-90af-3a9c59918e8a-tls-certs") pod "noten-model-a44263-kserve-5b4fc74fdf-6xcsb" (UID: "6d948091-5653-4673-90af-3a9c59918e8a") : secret "noten-model-a44263-kserve-self-signed-certs" not found
Nov 03 20:06:16.221303 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:06:16.221217    5591 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-self-signed-certs: secret "noten-model-a44263-kserve-self-signed-certs" not found
Nov 03 20:06:16.221303 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:06:16.221285    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d948091-5653-4673-90af-3a9c59918e8a-tls-certs podName:6d948091-5653-4673-90af-3a9c59918e8a nodeName:}" failed. No retries permitted until 2026-11-03 20:06:32.221270337 +0000 UTC m=+1740.993563346 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6d948091-5653-4673-90af-3a9c59918e8a-tls-certs") pod "noten-model-a44263-kserve-5b4fc74fdf-6xcsb" (UID: "6d948091-5653-4673-90af-3a9c59918e8a") : secret "noten-model-a44263-kserve-self-signed-certs" not found
Nov 03 20:06:16.344407 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:16.344373    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb" podUID="6d948091-5653-4673-90af-3a9c59918e8a" containerName="main" containerID="cri-o://34d78a278284814732969988a1c4c38789e553b2b497e15297f30a4a6ca6f5a3" gracePeriod=45
Nov 03 20:06:16.544689 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:16.544657    5591 generic.go:356] "Generic (PLEG): container finished" podID="6d948091-5653-4673-90af-3a9c59918e8a" containerID="34d78a278284814732969988a1c4c38789e553b2b497e15297f30a4a6ca6f5a3" exitCode=0
Nov 03 20:06:16.544898 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:16.544859    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb" event={"ID":"6d948091-5653-4673-90af-3a9c59918e8a","Type":"ContainerDied","Data":"34d78a278284814732969988a1c4c38789e553b2b497e15297f30a4a6ca6f5a3"}
Nov 03 20:06:16.699010 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:16.698578    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"
Nov 03 20:06:16.830676 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:16.830587    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-home\" (UniqueName: \"kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-home\") pod \"6d948091-5653-4673-90af-3a9c59918e8a\" (UID: \"6d948091-5653-4673-90af-3a9c59918e8a\") "
Nov 03 20:06:16.830676 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:16.830667    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-tmp-dir\") pod \"6d948091-5653-4673-90af-3a9c59918e8a\" (UID: \"6d948091-5653-4673-90af-3a9c59918e8a\") "
Nov 03 20:06:16.830871 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:16.830689    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d948091-5653-4673-90af-3a9c59918e8a-kube-api-access-lznw6\" (UniqueName: \"kubernetes.io/projected/6d948091-5653-4673-90af-3a9c59918e8a-kube-api-access-lznw6\") pod \"6d948091-5653-4673-90af-3a9c59918e8a\" (UID: \"6d948091-5653-4673-90af-3a9c59918e8a\") "
Nov 03 20:06:16.830871 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:16.830841    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-home" pod "6d948091-5653-4673-90af-3a9c59918e8a" (UID: "6d948091-5653-4673-90af-3a9c59918e8a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:16.830871 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:16.830863    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-dshm\") pod \"6d948091-5653-4673-90af-3a9c59918e8a\" (UID: \"6d948091-5653-4673-90af-3a9c59918e8a\") "
Nov 03 20:06:16.831043 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:16.830953    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d948091-5653-4673-90af-3a9c59918e8a-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d948091-5653-4673-90af-3a9c59918e8a-tls-certs\") pod \"6d948091-5653-4673-90af-3a9c59918e8a\" (UID: \"6d948091-5653-4673-90af-3a9c59918e8a\") "
Nov 03 20:06:16.831043 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:16.830965    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-tmp-dir" pod "6d948091-5653-4673-90af-3a9c59918e8a" (UID: "6d948091-5653-4673-90af-3a9c59918e8a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:16.831043 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:16.830983    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-model-cache\") pod \"6d948091-5653-4673-90af-3a9c59918e8a\" (UID: \"6d948091-5653-4673-90af-3a9c59918e8a\") "
Nov 03 20:06:16.831299 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:16.831278    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-model-cache" pod "6d948091-5653-4673-90af-3a9c59918e8a" (UID: "6d948091-5653-4673-90af-3a9c59918e8a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:16.832072 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:16.832051    5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-model-cache\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:16.832137 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:16.832080    5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-home\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:16.832137 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:16.832095    5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-tmp-dir\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:16.833175 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:16.833156    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d948091-5653-4673-90af-3a9c59918e8a-kube-api-access-lznw6" pod "6d948091-5653-4673-90af-3a9c59918e8a" (UID: "6d948091-5653-4673-90af-3a9c59918e8a"). InnerVolumeSpecName "kube-api-access-lznw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:16.833175 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:16.833160    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-dshm" pod "6d948091-5653-4673-90af-3a9c59918e8a" (UID: "6d948091-5653-4673-90af-3a9c59918e8a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:06:16.833363 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:16.833159    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d948091-5653-4673-90af-3a9c59918e8a-tls-certs" pod "6d948091-5653-4673-90af-3a9c59918e8a" (UID: "6d948091-5653-4673-90af-3a9c59918e8a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:06:16.934575 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:16.934541    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lznw6\" (UniqueName: \"kubernetes.io/projected/6d948091-5653-4673-90af-3a9c59918e8a-kube-api-access-lznw6\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:16.934575 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:16.934570    5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d948091-5653-4673-90af-3a9c59918e8a-dshm\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:16.934575 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:16.934579    5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d948091-5653-4673-90af-3a9c59918e8a-tls-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:17.554530 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:17.554498    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"
Nov 03 20:06:17.554530 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:17.554525    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb" event={"ID":"6d948091-5653-4673-90af-3a9c59918e8a","Type":"ContainerDied","Data":"2b34355b5b7804bbc86c9899b3ebb1f86b34b3d4f733c28edd19bf2998082f46"}
Nov 03 20:06:17.554993 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:17.554574    5591 scope.go:122] "RemoveContainer" containerID="34d78a278284814732969988a1c4c38789e553b2b497e15297f30a4a6ca6f5a3"
Nov 03 20:06:17.577442 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:17.577412    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"]
Nov 03 20:06:17.583734 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:17.583711    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-a44263/noten-model-a44263-kserve-5b4fc74fdf-6xcsb"]
Nov 03 20:06:19.352057 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:19.352026    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d948091-5653-4673-90af-3a9c59918e8a" path="/var/lib/kubelet/pods/6d948091-5653-4673-90af-3a9c59918e8a/volumes"
Nov 03 20:06:23.761873 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:23.761829    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:06:28.726248 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:06:28.726139    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:06:43.478858 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:43.478811    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7240-01af22"]
Nov 03 20:06:43.480050 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:43.480018    5591 cpu_manager.go:414] "containerMap: removing container" podUID="68b82f0c-e037-4366-891b-1ee898b2cba2" containerName="istio-proxy"
Nov 03 20:06:43.480050 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:43.480050    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68b82f0c-e037-4366-891b-1ee898b2cba2" containerName="istio-proxy"
Nov 03 20:06:43.480334 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:43.480104    5591 cpu_manager.go:414] "containerMap: removing container" podUID="1faa21e4-5bf7-418d-adc8-3fc5bf955010" containerName="istio-proxy"
Nov 03 20:06:43.480334 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:43.480113    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1faa21e4-5bf7-418d-adc8-3fc5bf955010" containerName="istio-proxy"
Nov 03 20:06:43.480334 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:43.480131    5591 cpu_manager.go:414] "containerMap: removing container" podUID="1faa21e4-5bf7-418d-adc8-3fc5bf955010" containerName="istio-proxy"
Nov 03 20:06:43.480334 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:43.480139    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1faa21e4-5bf7-418d-adc8-3fc5bf955010" containerName="istio-proxy"
Nov 03 20:06:43.480334 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:43.480186    5591 cpu_manager.go:414] "containerMap: removing container" podUID="6d948091-5653-4673-90af-3a9c59918e8a" containerName="main"
Nov 03 20:06:43.480334 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:43.480195    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d948091-5653-4673-90af-3a9c59918e8a" containerName="main"
Nov 03 20:06:43.480684 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:43.480426    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="68b82f0c-e037-4366-891b-1ee898b2cba2" containerName="istio-proxy"
Nov 03 20:06:43.480684 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:43.480451    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="1faa21e4-5bf7-418d-adc8-3fc5bf955010" containerName="istio-proxy"
Nov 03 20:06:43.480684 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:43.480462    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="1faa21e4-5bf7-418d-adc8-3fc5bf955010" containerName="istio-proxy"
Nov 03 20:06:43.480684 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:43.480478    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d948091-5653-4673-90af-3a9c59918e8a" containerName="main"
Nov 03 20:06:43.498155 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:43.498125    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-01af22"]
Nov 03 20:06:43.498494 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:43.498472    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-01af22"
Nov 03 20:06:43.503352 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:43.503328    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-tz7np\""
Nov 03 20:06:43.646473 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:43.646440    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8jdg\" (UniqueName: \"kubernetes.io/projected/2c1af614-fb83-4be4-8205-a95958ee5df3-kube-api-access-z8jdg\") pod \"test-curl-7240-01af22\" (UID: \"2c1af614-fb83-4be4-8205-a95958ee5df3\") " pod="opendatahub/test-curl-7240-01af22"
Nov 03 20:06:43.761293 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:43.760676    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8jdg\" (UniqueName: \"kubernetes.io/projected/2c1af614-fb83-4be4-8205-a95958ee5df3-kube-api-access-z8jdg\") pod \"test-curl-7240-01af22\" (UID: \"2c1af614-fb83-4be4-8205-a95958ee5df3\") " pod="opendatahub/test-curl-7240-01af22"
Nov 03 20:06:43.802354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:43.802278    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8jdg\" (UniqueName: \"kubernetes.io/projected/2c1af614-fb83-4be4-8205-a95958ee5df3-kube-api-access-z8jdg\") pod \"test-curl-7240-01af22\" (UID: \"2c1af614-fb83-4be4-8205-a95958ee5df3\") " pod="opendatahub/test-curl-7240-01af22"
Nov 03 20:06:43.848785 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:43.848751    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-01af22"
Nov 03 20:06:44.132875 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:44.132851    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-01af22"]
Nov 03 20:06:44.134586 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:06:44.134555    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c1af614_fb83_4be4_8205_a95958ee5df3.slice/crio-dd4bad6295cd7cebb0fb168b9bf13fe298054dcce048d274c73d90944dd929de WatchSource:0}: Error finding container dd4bad6295cd7cebb0fb168b9bf13fe298054dcce048d274c73d90944dd929de: Status 404 returned error can't find the container with id dd4bad6295cd7cebb0fb168b9bf13fe298054dcce048d274c73d90944dd929de
Nov 03 20:06:44.136623 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:44.136605    5591 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:06:44.833934 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:44.833900    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-01af22" event={"ID":"2c1af614-fb83-4be4-8205-a95958ee5df3","Type":"ContainerStarted","Data":"dd4bad6295cd7cebb0fb168b9bf13fe298054dcce048d274c73d90944dd929de"}
Nov 03 20:06:45.842926 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:45.842892    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-01af22" event={"ID":"2c1af614-fb83-4be4-8205-a95958ee5df3","Type":"ContainerStarted","Data":"870bae9d9460fbe1a02c58b10d4ec0f1f11d6590dd338820004fa67c505c41e8"}
Nov 03 20:06:45.861249 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:45.861177    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7240-01af22" podStartSLOduration=1.961557572 podStartE2EDuration="2.861162259s" totalImagesPullingTime="899.604687ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:45.860360648 +0000 UTC m=+1754.632653679" watchObservedRunningTime="2026-11-03 20:06:45.861162259 +0000 UTC m=+1754.633455277"
Nov 03 20:06:46.614199 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:46.614167    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7240-01af22"]
Nov 03 20:06:46.621323 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:46.621287    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7240-01af22"]
Nov 03 20:06:47.986405 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:47.986359    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7240-01af22" podUID="2c1af614-fb83-4be4-8205-a95958ee5df3" containerName="test-curl-7240-01af22" containerID="cri-o://870bae9d9460fbe1a02c58b10d4ec0f1f11d6590dd338820004fa67c505c41e8" gracePeriod=2
Nov 03 20:06:50.238716 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:50.238688    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-01af22"
Nov 03 20:06:50.310464 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:50.310363    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c1af614-fb83-4be4-8205-a95958ee5df3-kube-api-access-z8jdg\" (UniqueName: \"kubernetes.io/projected/2c1af614-fb83-4be4-8205-a95958ee5df3-kube-api-access-z8jdg\") pod \"2c1af614-fb83-4be4-8205-a95958ee5df3\" (UID: \"2c1af614-fb83-4be4-8205-a95958ee5df3\") "
Nov 03 20:06:50.313179 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:50.313145    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c1af614-fb83-4be4-8205-a95958ee5df3-kube-api-access-z8jdg" pod "2c1af614-fb83-4be4-8205-a95958ee5df3" (UID: "2c1af614-fb83-4be4-8205-a95958ee5df3"). InnerVolumeSpecName "kube-api-access-z8jdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:06:50.412613 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:50.412577    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8jdg\" (UniqueName: \"kubernetes.io/projected/2c1af614-fb83-4be4-8205-a95958ee5df3-kube-api-access-z8jdg\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:06:51.039612 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:51.039564    5591 generic.go:356] "Generic (PLEG): container finished" podID="2c1af614-fb83-4be4-8205-a95958ee5df3" containerID="870bae9d9460fbe1a02c58b10d4ec0f1f11d6590dd338820004fa67c505c41e8" exitCode=137
Nov 03 20:06:51.039796 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:51.039631    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-01af22"
Nov 03 20:06:51.039796 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:51.039640    5591 scope.go:122] "RemoveContainer" containerID="870bae9d9460fbe1a02c58b10d4ec0f1f11d6590dd338820004fa67c505c41e8"
Nov 03 20:06:51.050573 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:51.050557    5591 scope.go:122] "RemoveContainer" containerID="870bae9d9460fbe1a02c58b10d4ec0f1f11d6590dd338820004fa67c505c41e8"
Nov 03 20:06:51.050827 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:06:51.050805    5591 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-01af22\" is forbidden: User \"system:node:ip-10-0-143-206.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-206.us-east-2.compute.internal' and this object" podUID="2c1af614-fb83-4be4-8205-a95958ee5df3" pod="opendatahub/test-curl-7240-01af22"
Nov 03 20:06:51.050885 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:06:51.050828    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"870bae9d9460fbe1a02c58b10d4ec0f1f11d6590dd338820004fa67c505c41e8\": container with ID starting with 870bae9d9460fbe1a02c58b10d4ec0f1f11d6590dd338820004fa67c505c41e8 not found: ID does not exist" containerID="870bae9d9460fbe1a02c58b10d4ec0f1f11d6590dd338820004fa67c505c41e8"
Nov 03 20:06:51.050885 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:51.050858    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"870bae9d9460fbe1a02c58b10d4ec0f1f11d6590dd338820004fa67c505c41e8"} err="failed to get container status \"870bae9d9460fbe1a02c58b10d4ec0f1f11d6590dd338820004fa67c505c41e8\": rpc error: code = NotFound desc = could not find container \"870bae9d9460fbe1a02c58b10d4ec0f1f11d6590dd338820004fa67c505c41e8\": container with ID starting with 870bae9d9460fbe1a02c58b10d4ec0f1f11d6590dd338820004fa67c505c41e8 not found: ID does not exist"
Nov 03 20:06:51.348673 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:06:51.348583    5591 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-01af22\" is forbidden: User \"system:node:ip-10-0-143-206.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-206.us-east-2.compute.internal' and this object" podUID="2c1af614-fb83-4be4-8205-a95958ee5df3" pod="opendatahub/test-curl-7240-01af22"
Nov 03 20:06:51.350889 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:51.350866    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c1af614-fb83-4be4-8205-a95958ee5df3" path="/var/lib/kubelet/pods/2c1af614-fb83-4be4-8205-a95958ee5df3/volumes"
Nov 03 20:06:52.913428 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:52.913389    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd"]
Nov 03 20:06:52.914145 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:52.914131    5591 cpu_manager.go:414] "containerMap: removing container" podUID="2c1af614-fb83-4be4-8205-a95958ee5df3" containerName="test-curl-7240-01af22"
Nov 03 20:06:52.914191 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:52.914146    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c1af614-fb83-4be4-8205-a95958ee5df3" containerName="test-curl-7240-01af22"
Nov 03 20:06:52.914353 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:52.914339    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c1af614-fb83-4be4-8205-a95958ee5df3" containerName="test-curl-7240-01af22"
Nov 03 20:06:52.918099 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:52.918078    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd"
Nov 03 20:06:52.926424 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:52.926398    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-836265\""
Nov 03 20:06:52.930645 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:52.930615    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-836265-dockercfg-qmqcx\""
Nov 03 20:06:52.939612 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:52.939583    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd"]
Nov 03 20:06:52.965721 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:52.965685    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-568sw\" (UniqueName: \"kubernetes.io/projected/d26da545-16eb-415b-91ed-63db42e8f01d-kube-api-access-568sw\") pod \"payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd\" (UID: \"d26da545-16eb-415b-91ed-63db42e8f01d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd"
Nov 03 20:06:52.965880 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:52.965745    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d26da545-16eb-415b-91ed-63db42e8f01d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd\" (UID: \"d26da545-16eb-415b-91ed-63db42e8f01d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd"
Nov 03 20:06:53.110384 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:53.093290    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-568sw\" (UniqueName: \"kubernetes.io/projected/d26da545-16eb-415b-91ed-63db42e8f01d-kube-api-access-568sw\") pod \"payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd\" (UID: \"d26da545-16eb-415b-91ed-63db42e8f01d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd"
Nov 03 20:06:53.128125 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:53.128089    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d26da545-16eb-415b-91ed-63db42e8f01d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd\" (UID: \"d26da545-16eb-415b-91ed-63db42e8f01d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd"
Nov 03 20:06:53.146180 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:53.129280    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d26da545-16eb-415b-91ed-63db42e8f01d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd\" (UID: \"d26da545-16eb-415b-91ed-63db42e8f01d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd"
Nov 03 20:06:53.186617 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:53.186538    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-568sw\" (UniqueName: \"kubernetes.io/projected/d26da545-16eb-415b-91ed-63db42e8f01d-kube-api-access-568sw\") pod \"payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd\" (UID: \"d26da545-16eb-415b-91ed-63db42e8f01d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd"
Nov 03 20:06:53.240645 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:53.240610    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd"
Nov 03 20:06:53.542093 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:53.542066    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd"]
Nov 03 20:06:53.543303 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:06:53.543277    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26da545_16eb_415b_91ed_63db42e8f01d.slice/crio-6f8dd2d909a902830d4089d700d3a95222fcea9e9efdcd1961651c8f756c113e WatchSource:0}: Error finding container 6f8dd2d909a902830d4089d700d3a95222fcea9e9efdcd1961651c8f756c113e: Status 404 returned error can't find the container with id 6f8dd2d909a902830d4089d700d3a95222fcea9e9efdcd1961651c8f756c113e
Nov 03 20:06:54.205954 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:54.205920    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd" event={"ID":"d26da545-16eb-415b-91ed-63db42e8f01d","Type":"ContainerStarted","Data":"fa9e54a5b0138583280047135d5e5a3af0c2e887f2778e10ecf1a4800b290474"}
Nov 03 20:06:54.206661 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:54.205974    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd" event={"ID":"d26da545-16eb-415b-91ed-63db42e8f01d","Type":"ContainerStarted","Data":"6f8dd2d909a902830d4089d700d3a95222fcea9e9efdcd1961651c8f756c113e"}
Nov 03 20:06:54.206661 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:54.206007    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd"
Nov 03 20:06:54.223769 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:06:54.223458    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd" podStartSLOduration=2.223438935 podStartE2EDuration="2.223438935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:54.223378537 +0000 UTC m=+1762.995671556" watchObservedRunningTime="2026-11-03 20:06:54.223438935 +0000 UTC m=+1762.995731954"
Nov 03 20:06:58.728096 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:06:58.728049    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:07:00.849399 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:00.849343    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k"]
Nov 03 20:07:00.856667 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:00.856638    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k"
Nov 03 20:07:00.863409 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:00.863381    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-836265-serving-cert\""
Nov 03 20:07:00.869249 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:00.869197    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k"]
Nov 03 20:07:00.916688 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:00.916656    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k\" (UID: \"d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k"
Nov 03 20:07:00.916853 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:00.916732    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k\" (UID: \"d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k"
Nov 03 20:07:00.916853 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:00.916803    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqhxq\" (UniqueName: \"kubernetes.io/projected/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-kube-api-access-tqhxq\") pod \"maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k\" (UID: \"d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k"
Nov 03 20:07:00.916985 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:00.916970    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k\" (UID: \"d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k"
Nov 03 20:07:01.023228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:01.020166    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k\" (UID: \"d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k"
Nov 03 20:07:01.023228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:01.020406    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k\" (UID: \"d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k"
Nov 03 20:07:01.023228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:01.020543    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k\" (UID: \"d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k"
Nov 03 20:07:01.023228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:01.020592    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqhxq\" (UniqueName: \"kubernetes.io/projected/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-kube-api-access-tqhxq\") pod \"maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k\" (UID: \"d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k"
Nov 03 20:07:01.023228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:01.021927    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k\" (UID: \"d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k"
Nov 03 20:07:01.023228 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:07:01.022026    5591 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-serving-cert: secret "maas-api-e2e-shared-a-w2-836265-serving-cert" not found
Nov 03 20:07:01.023228 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:07:01.022089    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-maas-api-tls podName:d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb nodeName:}" failed. No retries permitted until 2026-11-03 20:07:01.522067097 +0000 UTC m=+1770.294360131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-maas-api-tls") pod "maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k" (UID: "d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb") : secret "maas-api-e2e-shared-a-w2-836265-serving-cert" not found
Nov 03 20:07:01.053272 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:01.047553    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k\" (UID: \"d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k"
Nov 03 20:07:01.054356 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:01.054022    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqhxq\" (UniqueName: \"kubernetes.io/projected/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-kube-api-access-tqhxq\") pod \"maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k\" (UID: \"d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k"
Nov 03 20:07:01.532233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:01.531453    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k\" (UID: \"d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k"
Nov 03 20:07:01.570235 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:01.566830    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k\" (UID: \"d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k"
Nov 03 20:07:01.785779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:01.785687    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k"
Nov 03 20:07:02.140276 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:02.136349    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k"]
Nov 03 20:07:02.344344 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:02.344308    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k" event={"ID":"d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb","Type":"ContainerStarted","Data":"3062b68738b2fe5d1e03d5c7898b15beb1bd117eefdbd4ed0f87a2a4504d9d51"}
Nov 03 20:07:03.352714 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:03.352664    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k" event={"ID":"d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb","Type":"ContainerStarted","Data":"401474caa9d654980d599143a885653f2c28b2a3b48d2025ad6fa707e252184f"}
Nov 03 20:07:03.353130 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:03.352743    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k"
Nov 03 20:07:03.377049 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:03.376996    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k" podStartSLOduration=2.882420875 podStartE2EDuration="3.376981899s" totalImagesPullingTime="494.561024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:03.372486531 +0000 UTC m=+1772.144779550" watchObservedRunningTime="2026-11-03 20:07:03.376981899 +0000 UTC m=+1772.149274914"
Nov 03 20:07:05.217245 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:05.217219    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd"
Nov 03 20:07:09.369939 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:09.369906    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k"
Nov 03 20:07:09.745723 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:09.745636    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:07:28.725771 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:07:28.725733    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:07:32.174041 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:32.174002    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6"]
Nov 03 20:07:32.179625 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:32.179597    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6"
Nov 03 20:07:32.188323 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:32.188297    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-660d95-serving-cert\""
Nov 03 20:07:32.191739 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:32.191519    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6"]
Nov 03 20:07:32.356734 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:32.356140    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df5c29d5-b63f-47ca-892a-d288dfa08f7f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6\" (UID: \"df5c29d5-b63f-47ca-892a-d288dfa08f7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6"
Nov 03 20:07:32.356734 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:32.356396    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vthcp\" (UniqueName: \"kubernetes.io/projected/df5c29d5-b63f-47ca-892a-d288dfa08f7f-kube-api-access-vthcp\") pod \"maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6\" (UID: \"df5c29d5-b63f-47ca-892a-d288dfa08f7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6"
Nov 03 20:07:32.356734 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:32.356477    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df5c29d5-b63f-47ca-892a-d288dfa08f7f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6\" (UID: \"df5c29d5-b63f-47ca-892a-d288dfa08f7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6"
Nov 03 20:07:32.356734 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:32.356536    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df5c29d5-b63f-47ca-892a-d288dfa08f7f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6\" (UID: \"df5c29d5-b63f-47ca-892a-d288dfa08f7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6"
Nov 03 20:07:32.457824 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:32.457726    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df5c29d5-b63f-47ca-892a-d288dfa08f7f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6\" (UID: \"df5c29d5-b63f-47ca-892a-d288dfa08f7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6"
Nov 03 20:07:32.457824 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:32.457790    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df5c29d5-b63f-47ca-892a-d288dfa08f7f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6\" (UID: \"df5c29d5-b63f-47ca-892a-d288dfa08f7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6"
Nov 03 20:07:32.458082 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:32.457940    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df5c29d5-b63f-47ca-892a-d288dfa08f7f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6\" (UID: \"df5c29d5-b63f-47ca-892a-d288dfa08f7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6"
Nov 03 20:07:32.458082 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:32.458027    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vthcp\" (UniqueName: \"kubernetes.io/projected/df5c29d5-b63f-47ca-892a-d288dfa08f7f-kube-api-access-vthcp\") pod \"maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6\" (UID: \"df5c29d5-b63f-47ca-892a-d288dfa08f7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6"
Nov 03 20:07:32.458718 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:32.458690    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df5c29d5-b63f-47ca-892a-d288dfa08f7f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6\" (UID: \"df5c29d5-b63f-47ca-892a-d288dfa08f7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6"
Nov 03 20:07:32.460543 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:32.460519    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df5c29d5-b63f-47ca-892a-d288dfa08f7f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6\" (UID: \"df5c29d5-b63f-47ca-892a-d288dfa08f7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6"
Nov 03 20:07:32.460623 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:32.460548    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df5c29d5-b63f-47ca-892a-d288dfa08f7f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6\" (UID: \"df5c29d5-b63f-47ca-892a-d288dfa08f7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6"
Nov 03 20:07:32.470543 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:32.470518    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vthcp\" (UniqueName: \"kubernetes.io/projected/df5c29d5-b63f-47ca-892a-d288dfa08f7f-kube-api-access-vthcp\") pod \"maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6\" (UID: \"df5c29d5-b63f-47ca-892a-d288dfa08f7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6"
Nov 03 20:07:32.500399 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:32.500354    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6"
Nov 03 20:07:32.865229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:32.859877    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6"]
Nov 03 20:07:33.523342 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:33.523316    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 20:07:33.528480 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:33.528451    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 20:07:33.681957 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:33.681870    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6" event={"ID":"df5c29d5-b63f-47ca-892a-d288dfa08f7f","Type":"ContainerStarted","Data":"37f1cfebbd3e07efd3ee56ec46169f142a939b80f299cd3d92ae9fed3cd0873b"}
Nov 03 20:07:33.681957 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:33.681929    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6" event={"ID":"df5c29d5-b63f-47ca-892a-d288dfa08f7f","Type":"ContainerStarted","Data":"cf688c99b4cfc0cebfe60d72403627d088345e1b9814994a77f5fb153086f986"}
Nov 03 20:07:33.681957 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:33.681950    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6"
Nov 03 20:07:33.709611 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:33.709556    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6" podStartSLOduration=1.187598778 podStartE2EDuration="1.709543248s" totalImagesPullingTime="521.94447ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:33.707044593 +0000 UTC m=+1802.479337615" watchObservedRunningTime="2026-11-03 20:07:33.709543248 +0000 UTC m=+1802.481836275"
Nov 03 20:07:39.695725 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:39.695686    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6"
Nov 03 20:07:52.255511 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:52.255437    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7240-e41344"]
Nov 03 20:07:52.260747 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:52.260728    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-e41344"
Nov 03 20:07:52.267284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:52.267261    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-e41344"]
Nov 03 20:07:52.340437 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:52.340406    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htgm7\" (UniqueName: \"kubernetes.io/projected/7c88a776-d0c3-42c8-801a-298f8560f09c-kube-api-access-htgm7\") pod \"test-curl-7240-e41344\" (UID: \"7c88a776-d0c3-42c8-801a-298f8560f09c\") " pod="opendatahub/test-curl-7240-e41344"
Nov 03 20:07:52.363343 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:52.363315    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-tz7np\""
Nov 03 20:07:52.441948 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:52.441912    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htgm7\" (UniqueName: \"kubernetes.io/projected/7c88a776-d0c3-42c8-801a-298f8560f09c-kube-api-access-htgm7\") pod \"test-curl-7240-e41344\" (UID: \"7c88a776-d0c3-42c8-801a-298f8560f09c\") " pod="opendatahub/test-curl-7240-e41344"
Nov 03 20:07:52.456471 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:52.456447    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htgm7\" (UniqueName: \"kubernetes.io/projected/7c88a776-d0c3-42c8-801a-298f8560f09c-kube-api-access-htgm7\") pod \"test-curl-7240-e41344\" (UID: \"7c88a776-d0c3-42c8-801a-298f8560f09c\") " pod="opendatahub/test-curl-7240-e41344"
Nov 03 20:07:52.579386 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:52.579352    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-e41344"
Nov 03 20:07:52.937114 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:52.937080    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7240-e41344"]
Nov 03 20:07:52.937725 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:07:52.937696    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c88a776_d0c3_42c8_801a_298f8560f09c.slice/crio-854aea24fc9ad4219e0a7b05c3bc0a9993055cffc8a336a26e7014c4c0b404e8 WatchSource:0}: Error finding container 854aea24fc9ad4219e0a7b05c3bc0a9993055cffc8a336a26e7014c4c0b404e8: Status 404 returned error can't find the container with id 854aea24fc9ad4219e0a7b05c3bc0a9993055cffc8a336a26e7014c4c0b404e8
Nov 03 20:07:53.857924 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:53.857877    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-e41344" event={"ID":"7c88a776-d0c3-42c8-801a-298f8560f09c","Type":"ContainerStarted","Data":"2d686f8379a11b07eee3d44cdbebf7f6a572fd1878749ec1d3c774152c6c8efa"}
Nov 03 20:07:53.858379 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:53.857933    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7240-e41344" event={"ID":"7c88a776-d0c3-42c8-801a-298f8560f09c","Type":"ContainerStarted","Data":"854aea24fc9ad4219e0a7b05c3bc0a9993055cffc8a336a26e7014c4c0b404e8"}
Nov 03 20:07:53.886497 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:53.886429    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7240-e41344" podStartSLOduration=1.8864103509999999 podStartE2EDuration="1.886410351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:07:53.881862577 +0000 UTC m=+1822.654155596" watchObservedRunningTime="2026-11-03 20:07:53.886410351 +0000 UTC m=+1822.658703374"
Nov 03 20:07:54.608384 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:54.605139    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7240-e41344"]
Nov 03 20:07:54.615082 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:54.615055    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7240-e41344"]
Nov 03 20:07:55.919689 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:55.919652    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7240-e41344" podUID="7c88a776-d0c3-42c8-801a-298f8560f09c" containerName="test-curl-7240-e41344" containerID="cri-o://2d686f8379a11b07eee3d44cdbebf7f6a572fd1878749ec1d3c774152c6c8efa" gracePeriod=2
Nov 03 20:07:58.216019 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:58.215991    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-e41344"
Nov 03 20:07:58.273380 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:58.273349    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c88a776-d0c3-42c8-801a-298f8560f09c-kube-api-access-htgm7\" (UniqueName: \"kubernetes.io/projected/7c88a776-d0c3-42c8-801a-298f8560f09c-kube-api-access-htgm7\") pod \"7c88a776-d0c3-42c8-801a-298f8560f09c\" (UID: \"7c88a776-d0c3-42c8-801a-298f8560f09c\") "
Nov 03 20:07:58.275641 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:58.275605    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c88a776-d0c3-42c8-801a-298f8560f09c-kube-api-access-htgm7" pod "7c88a776-d0c3-42c8-801a-298f8560f09c" (UID: "7c88a776-d0c3-42c8-801a-298f8560f09c"). InnerVolumeSpecName "kube-api-access-htgm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:07:58.375807 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:58.375766    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htgm7\" (UniqueName: \"kubernetes.io/projected/7c88a776-d0c3-42c8-801a-298f8560f09c-kube-api-access-htgm7\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:07:58.726722 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:07:58.726678    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:07:58.944371 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:58.944336    5591 generic.go:356] "Generic (PLEG): container finished" podID="7c88a776-d0c3-42c8-801a-298f8560f09c" containerID="2d686f8379a11b07eee3d44cdbebf7f6a572fd1878749ec1d3c774152c6c8efa" exitCode=137
Nov 03 20:07:58.944579 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:58.944388    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7240-e41344"
Nov 03 20:07:58.944579 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:58.944437    5591 scope.go:122] "RemoveContainer" containerID="2d686f8379a11b07eee3d44cdbebf7f6a572fd1878749ec1d3c774152c6c8efa"
Nov 03 20:07:58.957230 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:07:58.955624    5591 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7240-e41344\" is forbidden: User \"system:node:ip-10-0-143-206.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-206.us-east-2.compute.internal' and this object" podUID="7c88a776-d0c3-42c8-801a-298f8560f09c" pod="opendatahub/test-curl-7240-e41344"
Nov 03 20:07:58.958408 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:58.957845    5591 scope.go:122] "RemoveContainer" containerID="2d686f8379a11b07eee3d44cdbebf7f6a572fd1878749ec1d3c774152c6c8efa"
Nov 03 20:07:58.959367 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:07:58.959341    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d686f8379a11b07eee3d44cdbebf7f6a572fd1878749ec1d3c774152c6c8efa\": container with ID starting with 2d686f8379a11b07eee3d44cdbebf7f6a572fd1878749ec1d3c774152c6c8efa not found: ID does not exist" containerID="2d686f8379a11b07eee3d44cdbebf7f6a572fd1878749ec1d3c774152c6c8efa"
Nov 03 20:07:58.959478 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:58.959371    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d686f8379a11b07eee3d44cdbebf7f6a572fd1878749ec1d3c774152c6c8efa"} err="failed to get container status \"2d686f8379a11b07eee3d44cdbebf7f6a572fd1878749ec1d3c774152c6c8efa\": rpc error: code = NotFound desc = could not find container \"2d686f8379a11b07eee3d44cdbebf7f6a572fd1878749ec1d3c774152c6c8efa\": container with ID starting with 2d686f8379a11b07eee3d44cdbebf7f6a572fd1878749ec1d3c774152c6c8efa not found: ID does not exist"
Nov 03 20:07:59.351064 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:07:59.351027    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c88a776-d0c3-42c8-801a-298f8560f09c" path="/var/lib/kubelet/pods/7c88a776-d0c3-42c8-801a-298f8560f09c/volumes"
Nov 03 20:08:01.247896 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:01.247853    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:08:03.563028 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:03.562941    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"]
Nov 03 20:08:03.563474 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:03.563173    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47" podUID="ac8f5ae3-f947-438b-a93b-05bcdb362154" containerName="istio-proxy" containerID="cri-o://cbfec5d0839f38a2420161072e3f5588585ced998bf563158007ce7b4d35d833" gracePeriod=30
Nov 03 20:08:08.915301 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:08.915270    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:08:09.014762 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.014722    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-token\" (UniqueName: \"kubernetes.io/projected/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-token\") pod \"ac8f5ae3-f947-438b-a93b-05bcdb362154\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") "
Nov 03 20:08:09.014962 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.014773    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac8f5ae3-f947-438b-a93b-05bcdb362154-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac8f5ae3-f947-438b-a93b-05bcdb362154-istiod-ca-cert\") pod \"ac8f5ae3-f947-438b-a93b-05bcdb362154\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") "
Nov 03 20:08:09.014962 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.014810    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-workload-socket\") pod \"ac8f5ae3-f947-438b-a93b-05bcdb362154\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") "
Nov 03 20:08:09.014962 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.014942    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac8f5ae3-f947-438b-a93b-05bcdb362154-kube-api-access-qr6s4\" (UniqueName: \"kubernetes.io/projected/ac8f5ae3-f947-438b-a93b-05bcdb362154-kube-api-access-qr6s4\") pod \"ac8f5ae3-f947-438b-a93b-05bcdb362154\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") "
Nov 03 20:08:09.015125 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.014990    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-workload-certs\") pod \"ac8f5ae3-f947-438b-a93b-05bcdb362154\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") "
Nov 03 20:08:09.015125 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.015037    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-credential-socket\") pod \"ac8f5ae3-f947-438b-a93b-05bcdb362154\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") "
Nov 03 20:08:09.015125 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.015109    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-envoy\") pod \"ac8f5ae3-f947-438b-a93b-05bcdb362154\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") "
Nov 03 20:08:09.015369 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.015148    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-data\") pod \"ac8f5ae3-f947-438b-a93b-05bcdb362154\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") "
Nov 03 20:08:09.015369 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.015345    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-podinfo\") pod \"ac8f5ae3-f947-438b-a93b-05bcdb362154\" (UID: \"ac8f5ae3-f947-438b-a93b-05bcdb362154\") "
Nov 03 20:08:09.015480 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.015396    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-credential-socket" pod "ac8f5ae3-f947-438b-a93b-05bcdb362154" (UID: "ac8f5ae3-f947-438b-a93b-05bcdb362154"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:09.015480 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.015435    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac8f5ae3-f947-438b-a93b-05bcdb362154-istiod-ca-cert" pod "ac8f5ae3-f947-438b-a93b-05bcdb362154" (UID: "ac8f5ae3-f947-438b-a93b-05bcdb362154"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:08:09.015480 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.015445    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-workload-socket" pod "ac8f5ae3-f947-438b-a93b-05bcdb362154" (UID: "ac8f5ae3-f947-438b-a93b-05bcdb362154"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:09.015640 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.015620    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-data" pod "ac8f5ae3-f947-438b-a93b-05bcdb362154" (UID: "ac8f5ae3-f947-438b-a93b-05bcdb362154"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:09.015929 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.015899    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-workload-certs" pod "ac8f5ae3-f947-438b-a93b-05bcdb362154" (UID: "ac8f5ae3-f947-438b-a93b-05bcdb362154"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:09.016361 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.016289    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:09.016361 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.016316    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac8f5ae3-f947-438b-a93b-05bcdb362154-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:09.016361 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.016332    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:09.016361 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.016346    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:09.016361 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.016359    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:09.017551 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.017525    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac8f5ae3-f947-438b-a93b-05bcdb362154-kube-api-access-qr6s4" pod "ac8f5ae3-f947-438b-a93b-05bcdb362154" (UID: "ac8f5ae3-f947-438b-a93b-05bcdb362154"). InnerVolumeSpecName "kube-api-access-qr6s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:09.017663 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.017569    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-token" pod "ac8f5ae3-f947-438b-a93b-05bcdb362154" (UID: "ac8f5ae3-f947-438b-a93b-05bcdb362154"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:09.017923 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.017909    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-podinfo" pod "ac8f5ae3-f947-438b-a93b-05bcdb362154" (UID: "ac8f5ae3-f947-438b-a93b-05bcdb362154"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:08:09.018124 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.018111    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-envoy" pod "ac8f5ae3-f947-438b-a93b-05bcdb362154" (UID: "ac8f5ae3-f947-438b-a93b-05bcdb362154"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:09.025812 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.025777    5591 generic.go:356] "Generic (PLEG): container finished" podID="ac8f5ae3-f947-438b-a93b-05bcdb362154" containerID="cbfec5d0839f38a2420161072e3f5588585ced998bf563158007ce7b4d35d833" exitCode=0
Nov 03 20:08:09.025931 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.025825    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"
Nov 03 20:08:09.025931 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.025857    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47" event={"ID":"ac8f5ae3-f947-438b-a93b-05bcdb362154","Type":"ContainerDied","Data":"cbfec5d0839f38a2420161072e3f5588585ced998bf563158007ce7b4d35d833"}
Nov 03 20:08:09.025931 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.025913    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47" event={"ID":"ac8f5ae3-f947-438b-a93b-05bcdb362154","Type":"ContainerDied","Data":"454765e2179b815c6068fee12087a79a07eda47f05a376c095d6da04852b1638"}
Nov 03 20:08:09.026061 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.025936    5591 scope.go:122] "RemoveContainer" containerID="cbfec5d0839f38a2420161072e3f5588585ced998bf563158007ce7b4d35d833"
Nov 03 20:08:09.065419 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.065383    5591 scope.go:122] "RemoveContainer" containerID="cbfec5d0839f38a2420161072e3f5588585ced998bf563158007ce7b4d35d833"
Nov 03 20:08:09.065826 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:08:09.065798    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbfec5d0839f38a2420161072e3f5588585ced998bf563158007ce7b4d35d833\": container with ID starting with cbfec5d0839f38a2420161072e3f5588585ced998bf563158007ce7b4d35d833 not found: ID does not exist" containerID="cbfec5d0839f38a2420161072e3f5588585ced998bf563158007ce7b4d35d833"
Nov 03 20:08:09.065937 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.065838    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbfec5d0839f38a2420161072e3f5588585ced998bf563158007ce7b4d35d833"} err="failed to get container status \"cbfec5d0839f38a2420161072e3f5588585ced998bf563158007ce7b4d35d833\": rpc error: code = NotFound desc = could not find container \"cbfec5d0839f38a2420161072e3f5588585ced998bf563158007ce7b4d35d833\": container with ID starting with cbfec5d0839f38a2420161072e3f5588585ced998bf563158007ce7b4d35d833 not found: ID does not exist"
Nov 03 20:08:09.069286 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.069261    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"]
Nov 03 20:08:09.072898 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.072877    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-7kx47"]
Nov 03 20:08:09.117745 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.117669    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr6s4\" (UniqueName: \"kubernetes.io/projected/ac8f5ae3-f947-438b-a93b-05bcdb362154-kube-api-access-qr6s4\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:09.117745 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.117697    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:09.117745 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.117709    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:09.117745 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.117717    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac8f5ae3-f947-438b-a93b-05bcdb362154-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:09.351092 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:09.351054    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac8f5ae3-f947-438b-a93b-05bcdb362154" path="/var/lib/kubelet/pods/ac8f5ae3-f947-438b-a93b-05bcdb362154/volumes"
Nov 03 20:08:17.531971 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.531926    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc"]
Nov 03 20:08:17.532775 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.532760    5591 cpu_manager.go:414] "containerMap: removing container" podUID="ac8f5ae3-f947-438b-a93b-05bcdb362154" containerName="istio-proxy"
Nov 03 20:08:17.532833 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.532778    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac8f5ae3-f947-438b-a93b-05bcdb362154" containerName="istio-proxy"
Nov 03 20:08:17.532833 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.532830    5591 cpu_manager.go:414] "containerMap: removing container" podUID="7c88a776-d0c3-42c8-801a-298f8560f09c" containerName="test-curl-7240-e41344"
Nov 03 20:08:17.532903 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.532836    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c88a776-d0c3-42c8-801a-298f8560f09c" containerName="test-curl-7240-e41344"
Nov 03 20:08:17.532951 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.532941    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c88a776-d0c3-42c8-801a-298f8560f09c" containerName="test-curl-7240-e41344"
Nov 03 20:08:17.532989 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.532955    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac8f5ae3-f947-438b-a93b-05bcdb362154" containerName="istio-proxy"
Nov 03 20:08:17.538394 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.538373    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc"
Nov 03 20:08:17.545121 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.545094    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc"]
Nov 03 20:08:17.552604 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.552482    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-eb69cb\""
Nov 03 20:08:17.722229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.719940    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgtmb\" (UniqueName: \"kubernetes.io/projected/b51ae7c4-0791-4250-809e-b558456a410b-kube-api-access-wgtmb\") pod \"payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc\" (UID: \"b51ae7c4-0791-4250-809e-b558456a410b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc"
Nov 03 20:08:17.722229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.720245    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b51ae7c4-0791-4250-809e-b558456a410b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc\" (UID: \"b51ae7c4-0791-4250-809e-b558456a410b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc"
Nov 03 20:08:17.805359 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.804311    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"]
Nov 03 20:08:17.830230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.821342    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b51ae7c4-0791-4250-809e-b558456a410b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc\" (UID: \"b51ae7c4-0791-4250-809e-b558456a410b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc"
Nov 03 20:08:17.830230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.821497    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgtmb\" (UniqueName: \"kubernetes.io/projected/b51ae7c4-0791-4250-809e-b558456a410b-kube-api-access-wgtmb\") pod \"payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc\" (UID: \"b51ae7c4-0791-4250-809e-b558456a410b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc"
Nov 03 20:08:17.830230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.822890    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b51ae7c4-0791-4250-809e-b558456a410b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc\" (UID: \"b51ae7c4-0791-4250-809e-b558456a410b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc"
Nov 03 20:08:17.843487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.843459    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:17.883243 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.845869    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"]
Nov 03 20:08:17.883526 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.858861    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgtmb\" (UniqueName: \"kubernetes.io/projected/b51ae7c4-0791-4250-809e-b558456a410b-kube-api-access-wgtmb\") pod \"payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc\" (UID: \"b51ae7c4-0791-4250-809e-b558456a410b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc"
Nov 03 20:08:17.883657 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.873469    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-660d95-openshift-default-dockercfg-df2mh\""
Nov 03 20:08:17.941521 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.941475    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e760329-c9c8-4d93-8efb-8aaa938899b5-istiod-ca-cert\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:17.941712 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.941603    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-podinfo\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:17.941712 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.941632    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-data\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:17.941817 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.941792    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-workload-certs\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:17.941931 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.941910    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-token\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:17.942013 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.941967    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-credential-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:17.942069 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.942027    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-envoy\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:17.942142 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.942126    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p89x\" (UniqueName: \"kubernetes.io/projected/7e760329-c9c8-4d93-8efb-8aaa938899b5-kube-api-access-5p89x\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:17.942218 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:17.942162    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-workload-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:18.043368 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:18.043335    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-podinfo\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:18.043548 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:18.043375    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-data\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:18.043548 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:18.043466    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-workload-certs\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:18.043548 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:18.043533    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-token\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:18.043716 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:18.043573    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-credential-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:18.043716 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:18.043625    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-envoy\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:18.043818 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:18.043726    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p89x\" (UniqueName: \"kubernetes.io/projected/7e760329-c9c8-4d93-8efb-8aaa938899b5-kube-api-access-5p89x\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:18.043818 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:18.043762    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-workload-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:18.043955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:18.043865    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e760329-c9c8-4d93-8efb-8aaa938899b5-istiod-ca-cert\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:18.043955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:18.043884    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-data\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:18.043955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:18.043933    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-workload-certs\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:18.044137 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:18.043971    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-credential-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:18.044279 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:18.044257    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-workload-socket\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:18.044444 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:18.044427    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e760329-c9c8-4d93-8efb-8aaa938899b5-istiod-ca-cert\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:18.046193 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:18.046168    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-envoy\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:18.046338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:18.046194    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-podinfo\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:18.059334 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:18.059264    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-token\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:18.059334 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:18.059267    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p89x\" (UniqueName: \"kubernetes.io/projected/7e760329-c9c8-4d93-8efb-8aaa938899b5-kube-api-access-5p89x\") pod \"e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") " pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:18.162006 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:18.161970    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc"
Nov 03 20:08:18.257372 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:18.257034    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:18.450299 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:18.450262    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc"]
Nov 03 20:08:18.451046 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:08:18.451020    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb51ae7c4_0791_4250_809e_b558456a410b.slice/crio-e50822554f8b261ba8e82353afe34de9fad4c935915d9d803fc8b9ed1b3d24db WatchSource:0}: Error finding container e50822554f8b261ba8e82353afe34de9fad4c935915d9d803fc8b9ed1b3d24db: Status 404 returned error can't find the container with id e50822554f8b261ba8e82353afe34de9fad4c935915d9d803fc8b9ed1b3d24db
Nov 03 20:08:18.566529 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:18.566500    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"]
Nov 03 20:08:18.568282 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:08:18.568252    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e760329_c9c8_4d93_8efb_8aaa938899b5.slice/crio-312aeb20a2d5993190317597d1bdb36d5c1b86763616247af6b75055d14af58b WatchSource:0}: Error finding container 312aeb20a2d5993190317597d1bdb36d5c1b86763616247af6b75055d14af58b: Status 404 returned error can't find the container with id 312aeb20a2d5993190317597d1bdb36d5c1b86763616247af6b75055d14af58b
Nov 03 20:08:18.571471 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:18.571433    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:08:18.571540 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:18.571517    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:08:18.571584 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:18.571560    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:08:19.145641 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:19.145127    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc" event={"ID":"b51ae7c4-0791-4250-809e-b558456a410b","Type":"ContainerStarted","Data":"d101cb468d5e4c2a36a0875ad87b74fc242e2c92cc5ff901637169249d3f9e94"}
Nov 03 20:08:19.145641 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:19.145196    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc" event={"ID":"b51ae7c4-0791-4250-809e-b558456a410b","Type":"ContainerStarted","Data":"e50822554f8b261ba8e82353afe34de9fad4c935915d9d803fc8b9ed1b3d24db"}
Nov 03 20:08:19.146078 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:19.146060    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc"
Nov 03 20:08:19.148476 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:19.148449    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv" event={"ID":"7e760329-c9c8-4d93-8efb-8aaa938899b5","Type":"ContainerStarted","Data":"7723978d5daa9201a3078a7e34e842e0b4c794ef1610927634de33ff2c465e3e"}
Nov 03 20:08:19.148700 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:19.148683    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv" event={"ID":"7e760329-c9c8-4d93-8efb-8aaa938899b5","Type":"ContainerStarted","Data":"312aeb20a2d5993190317597d1bdb36d5c1b86763616247af6b75055d14af58b"}
Nov 03 20:08:19.166585 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:19.166542    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc" podStartSLOduration=2.166529785 podStartE2EDuration="2.166529785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:19.16502674 +0000 UTC m=+1847.937319761" watchObservedRunningTime="2026-11-03 20:08:19.166529785 +0000 UTC m=+1847.938822803"
Nov 03 20:08:19.195289 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:19.195216    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv" podStartSLOduration=2.19518548 podStartE2EDuration="2.19518548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:19.191095651 +0000 UTC m=+1847.963388663" watchObservedRunningTime="2026-11-03 20:08:19.19518548 +0000 UTC m=+1847.967478500"
Nov 03 20:08:19.258222 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:19.258170    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:19.259489 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:19.259462    5591 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:08:19.259627 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:19.259509    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv" podUID="7e760329-c9c8-4d93-8efb-8aaa938899b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 03 20:08:20.263488 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:20.263452    5591 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:08:20.264113 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:20.264080    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv" podUID="7e760329-c9c8-4d93-8efb-8aaa938899b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 03 20:08:21.257971 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:21.257934    5591 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:08:21.258176 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:21.258014    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv" podUID="7e760329-c9c8-4d93-8efb-8aaa938899b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 03 20:08:22.258132 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:22.258091    5591 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:08:22.258590 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:22.258151    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv" podUID="7e760329-c9c8-4d93-8efb-8aaa938899b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 03 20:08:23.258127 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:23.258091    5591 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:08:23.258334 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:23.258150    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv" podUID="7e760329-c9c8-4d93-8efb-8aaa938899b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 03 20:08:24.258682 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:24.258624    5591 patch_prober.go:29] interesting pod/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 03 20:08:24.259583 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:24.258721    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv" podUID="7e760329-c9c8-4d93-8efb-8aaa938899b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 03 20:08:25.263758 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:25.263729    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:25.264229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:25.264042    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:25.264970 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:25.264949    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:08:28.728624 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:08:28.728572    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:08:31.265382 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:31.265339    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc"
Nov 03 20:08:36.128464 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:36.128428    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7237-99891e"]
Nov 03 20:08:36.135185 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:36.135155    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7237-99891e"
Nov 03 20:08:36.139747 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:36.139719    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-tz7np\""
Nov 03 20:08:36.139970 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:36.139947    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7237-99891e"]
Nov 03 20:08:36.210072 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:36.210036    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8c86\" (UniqueName: \"kubernetes.io/projected/28e65dfe-a7e4-466f-a828-bd4d5900afed-kube-api-access-c8c86\") pod \"test-curl-7237-99891e\" (UID: \"28e65dfe-a7e4-466f-a828-bd4d5900afed\") " pod="opendatahub/test-curl-7237-99891e"
Nov 03 20:08:36.311958 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:36.311923    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8c86\" (UniqueName: \"kubernetes.io/projected/28e65dfe-a7e4-466f-a828-bd4d5900afed-kube-api-access-c8c86\") pod \"test-curl-7237-99891e\" (UID: \"28e65dfe-a7e4-466f-a828-bd4d5900afed\") " pod="opendatahub/test-curl-7237-99891e"
Nov 03 20:08:36.326714 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:36.326682    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8c86\" (UniqueName: \"kubernetes.io/projected/28e65dfe-a7e4-466f-a828-bd4d5900afed-kube-api-access-c8c86\") pod \"test-curl-7237-99891e\" (UID: \"28e65dfe-a7e4-466f-a828-bd4d5900afed\") " pod="opendatahub/test-curl-7237-99891e"
Nov 03 20:08:36.452558 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:36.452526    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7237-99891e"
Nov 03 20:08:36.748870 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:36.748842    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7237-99891e"]
Nov 03 20:08:36.749490 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:08:36.749462    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28e65dfe_a7e4_466f_a828_bd4d5900afed.slice/crio-d17dd1f1ceb76f7726b21bc90e98232b5b6df4cfcbcbd49c0071b65c4ad9d424 WatchSource:0}: Error finding container d17dd1f1ceb76f7726b21bc90e98232b5b6df4cfcbcbd49c0071b65c4ad9d424: Status 404 returned error can't find the container with id d17dd1f1ceb76f7726b21bc90e98232b5b6df4cfcbcbd49c0071b65c4ad9d424
Nov 03 20:08:37.611066 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:37.611022    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7237-99891e" event={"ID":"28e65dfe-a7e4-466f-a828-bd4d5900afed","Type":"ContainerStarted","Data":"ee200392e1f4f1ff7dcfde53d0a404c27d8b46b7ee9ff46364356a92645d3de0"}
Nov 03 20:08:37.611670 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:37.611085    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7237-99891e" event={"ID":"28e65dfe-a7e4-466f-a828-bd4d5900afed","Type":"ContainerStarted","Data":"d17dd1f1ceb76f7726b21bc90e98232b5b6df4cfcbcbd49c0071b65c4ad9d424"}
Nov 03 20:08:37.629417 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:37.629365    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7237-99891e" podStartSLOduration=1.629348351 podStartE2EDuration="1.629348351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:37.627551921 +0000 UTC m=+1866.399844940" watchObservedRunningTime="2026-11-03 20:08:37.629348351 +0000 UTC m=+1866.401641369"
Nov 03 20:08:38.496932 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:38.496886    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7237-99891e"]
Nov 03 20:08:38.501046 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:38.501016    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7237-99891e"]
Nov 03 20:08:39.626613 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:39.626554    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7237-99891e" podUID="28e65dfe-a7e4-466f-a828-bd4d5900afed" containerName="test-curl-7237-99891e" containerID="cri-o://ee200392e1f4f1ff7dcfde53d0a404c27d8b46b7ee9ff46364356a92645d3de0" gracePeriod=2
Nov 03 20:08:41.885590 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:41.885526    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7237-99891e"
Nov 03 20:08:41.925987 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:41.925953    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28e65dfe-a7e4-466f-a828-bd4d5900afed-kube-api-access-c8c86\" (UniqueName: \"kubernetes.io/projected/28e65dfe-a7e4-466f-a828-bd4d5900afed-kube-api-access-c8c86\") pod \"28e65dfe-a7e4-466f-a828-bd4d5900afed\" (UID: \"28e65dfe-a7e4-466f-a828-bd4d5900afed\") "
Nov 03 20:08:41.928354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:41.928323    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e65dfe-a7e4-466f-a828-bd4d5900afed-kube-api-access-c8c86" pod "28e65dfe-a7e4-466f-a828-bd4d5900afed" (UID: "28e65dfe-a7e4-466f-a828-bd4d5900afed"). InnerVolumeSpecName "kube-api-access-c8c86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:42.037754 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:42.037695    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8c86\" (UniqueName: \"kubernetes.io/projected/28e65dfe-a7e4-466f-a828-bd4d5900afed-kube-api-access-c8c86\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:42.665169 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:42.665128    5591 generic.go:356] "Generic (PLEG): container finished" podID="28e65dfe-a7e4-466f-a828-bd4d5900afed" containerID="ee200392e1f4f1ff7dcfde53d0a404c27d8b46b7ee9ff46364356a92645d3de0" exitCode=137
Nov 03 20:08:42.665169 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:42.665172    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7237-99891e"
Nov 03 20:08:42.665407 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:42.665248    5591 scope.go:122] "RemoveContainer" containerID="ee200392e1f4f1ff7dcfde53d0a404c27d8b46b7ee9ff46364356a92645d3de0"
Nov 03 20:08:42.678665 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:08:42.678633    5591 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7237-99891e\" is forbidden: User \"system:node:ip-10-0-143-206.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-206.us-east-2.compute.internal' and this object" podUID="28e65dfe-a7e4-466f-a828-bd4d5900afed" pod="opendatahub/test-curl-7237-99891e"
Nov 03 20:08:42.678936 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:42.678916    5591 scope.go:122] "RemoveContainer" containerID="ee200392e1f4f1ff7dcfde53d0a404c27d8b46b7ee9ff46364356a92645d3de0"
Nov 03 20:08:42.679289 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:08:42.679270    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee200392e1f4f1ff7dcfde53d0a404c27d8b46b7ee9ff46364356a92645d3de0\": container with ID starting with ee200392e1f4f1ff7dcfde53d0a404c27d8b46b7ee9ff46364356a92645d3de0 not found: ID does not exist" containerID="ee200392e1f4f1ff7dcfde53d0a404c27d8b46b7ee9ff46364356a92645d3de0"
Nov 03 20:08:42.679366 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:42.679298    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee200392e1f4f1ff7dcfde53d0a404c27d8b46b7ee9ff46364356a92645d3de0"} err="failed to get container status \"ee200392e1f4f1ff7dcfde53d0a404c27d8b46b7ee9ff46364356a92645d3de0\": rpc error: code = NotFound desc = could not find container \"ee200392e1f4f1ff7dcfde53d0a404c27d8b46b7ee9ff46364356a92645d3de0\": container with ID starting with ee200392e1f4f1ff7dcfde53d0a404c27d8b46b7ee9ff46364356a92645d3de0 not found: ID does not exist"
Nov 03 20:08:43.241152 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:43.241113    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"]
Nov 03 20:08:43.269853 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:08:43.269811    5591 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7237-99891e\" is forbidden: User \"system:node:ip-10-0-143-206.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-206.us-east-2.compute.internal' and this object" podUID="28e65dfe-a7e4-466f-a828-bd4d5900afed" pod="opendatahub/test-curl-7237-99891e"
Nov 03 20:08:43.415672 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:43.415624    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28e65dfe-a7e4-466f-a828-bd4d5900afed" path="/var/lib/kubelet/pods/28e65dfe-a7e4-466f-a828-bd4d5900afed/volumes"
Nov 03 20:08:52.959687 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:52.959651    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27"]
Nov 03 20:08:52.960577 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:52.960444    5591 cpu_manager.go:414] "containerMap: removing container" podUID="28e65dfe-a7e4-466f-a828-bd4d5900afed" containerName="test-curl-7237-99891e"
Nov 03 20:08:52.960577 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:52.960457    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28e65dfe-a7e4-466f-a828-bd4d5900afed" containerName="test-curl-7237-99891e"
Nov 03 20:08:52.960690 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:52.960621    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="28e65dfe-a7e4-466f-a828-bd4d5900afed" containerName="test-curl-7237-99891e"
Nov 03 20:08:52.964261 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:52.964240    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27"
Nov 03 20:08:52.970975 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:52.970953    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-8b7351\""
Nov 03 20:08:52.975676 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:52.975652    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27"]
Nov 03 20:08:52.982407 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:52.982378    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snvgp\" (UniqueName: \"kubernetes.io/projected/a67dbe17-4521-40e5-b4aa-602094676670-kube-api-access-snvgp\") pod \"payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27\" (UID: \"a67dbe17-4521-40e5-b4aa-602094676670\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27"
Nov 03 20:08:52.982836 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:52.982814    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67dbe17-4521-40e5-b4aa-602094676670-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27\" (UID: \"a67dbe17-4521-40e5-b4aa-602094676670\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27"
Nov 03 20:08:53.002659 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:53.002619    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm"]
Nov 03 20:08:53.008188 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:53.008159    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm"
Nov 03 20:08:53.019338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:53.019309    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm"]
Nov 03 20:08:53.019684 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:53.019645    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-8b7351-dockercfg-9bp4p\""
Nov 03 20:08:53.084041 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:53.084007    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snvgp\" (UniqueName: \"kubernetes.io/projected/a67dbe17-4521-40e5-b4aa-602094676670-kube-api-access-snvgp\") pod \"payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27\" (UID: \"a67dbe17-4521-40e5-b4aa-602094676670\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27"
Nov 03 20:08:53.084287 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:53.084164    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c2m5\" (UniqueName: \"kubernetes.io/projected/fe2c855d-4f89-4546-b580-e5f7e22eb9bc-kube-api-access-6c2m5\") pod \"payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm\" (UID: \"fe2c855d-4f89-4546-b580-e5f7e22eb9bc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm"
Nov 03 20:08:53.084287 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:53.084215    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2c855d-4f89-4546-b580-e5f7e22eb9bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm\" (UID: \"fe2c855d-4f89-4546-b580-e5f7e22eb9bc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm"
Nov 03 20:08:53.084470 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:53.084449    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67dbe17-4521-40e5-b4aa-602094676670-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27\" (UID: \"a67dbe17-4521-40e5-b4aa-602094676670\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27"
Nov 03 20:08:53.085373 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:53.085347    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67dbe17-4521-40e5-b4aa-602094676670-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27\" (UID: \"a67dbe17-4521-40e5-b4aa-602094676670\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27"
Nov 03 20:08:53.103616 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:53.098930    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snvgp\" (UniqueName: \"kubernetes.io/projected/a67dbe17-4521-40e5-b4aa-602094676670-kube-api-access-snvgp\") pod \"payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27\" (UID: \"a67dbe17-4521-40e5-b4aa-602094676670\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27"
Nov 03 20:08:53.188227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:53.187353    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c2m5\" (UniqueName: \"kubernetes.io/projected/fe2c855d-4f89-4546-b580-e5f7e22eb9bc-kube-api-access-6c2m5\") pod \"payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm\" (UID: \"fe2c855d-4f89-4546-b580-e5f7e22eb9bc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm"
Nov 03 20:08:53.188227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:53.187415    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2c855d-4f89-4546-b580-e5f7e22eb9bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm\" (UID: \"fe2c855d-4f89-4546-b580-e5f7e22eb9bc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm"
Nov 03 20:08:53.196229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:53.188673    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2c855d-4f89-4546-b580-e5f7e22eb9bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm\" (UID: \"fe2c855d-4f89-4546-b580-e5f7e22eb9bc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm"
Nov 03 20:08:53.231279 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:53.231024    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c2m5\" (UniqueName: \"kubernetes.io/projected/fe2c855d-4f89-4546-b580-e5f7e22eb9bc-kube-api-access-6c2m5\") pod \"payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm\" (UID: \"fe2c855d-4f89-4546-b580-e5f7e22eb9bc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm"
Nov 03 20:08:53.296231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:53.295607    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27"
Nov 03 20:08:53.332999 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:53.332942    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm"
Nov 03 20:08:53.646699 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:53.646670    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27"]
Nov 03 20:08:53.649992 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:08:53.649510    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67dbe17_4521_40e5_b4aa_602094676670.slice/crio-6489c477db794c5c1819716e21375e2dc49786251f28b44d39155edccf3870d9 WatchSource:0}: Error finding container 6489c477db794c5c1819716e21375e2dc49786251f28b44d39155edccf3870d9: Status 404 returned error can't find the container with id 6489c477db794c5c1819716e21375e2dc49786251f28b44d39155edccf3870d9
Nov 03 20:08:53.858458 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:53.858428    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm"]
Nov 03 20:08:53.858597 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:08:53.858577    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe2c855d_4f89_4546_b580_e5f7e22eb9bc.slice/crio-c83117b5d799e654d4c3c14fd1ee15149d618c0d71d457540f10cff89b118c62 WatchSource:0}: Error finding container c83117b5d799e654d4c3c14fd1ee15149d618c0d71d457540f10cff89b118c62: Status 404 returned error can't find the container with id c83117b5d799e654d4c3c14fd1ee15149d618c0d71d457540f10cff89b118c62
Nov 03 20:08:53.923394 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:53.923352    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27" event={"ID":"a67dbe17-4521-40e5-b4aa-602094676670","Type":"ContainerStarted","Data":"58ef0ded3606a7fd1d51409e3e3fb3e23260372f4fb4fdee41f07e7888de98d1"}
Nov 03 20:08:53.923563 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:53.923440    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27"
Nov 03 20:08:53.923563 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:53.923467    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27" event={"ID":"a67dbe17-4521-40e5-b4aa-602094676670","Type":"ContainerStarted","Data":"6489c477db794c5c1819716e21375e2dc49786251f28b44d39155edccf3870d9"}
Nov 03 20:08:53.924865 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:53.924840    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm" event={"ID":"fe2c855d-4f89-4546-b580-e5f7e22eb9bc","Type":"ContainerStarted","Data":"c83117b5d799e654d4c3c14fd1ee15149d618c0d71d457540f10cff89b118c62"}
Nov 03 20:08:53.943725 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:53.943668    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27" podStartSLOduration=1.943651706 podStartE2EDuration="1.943651706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:53.942429051 +0000 UTC m=+1882.714722070" watchObservedRunningTime="2026-11-03 20:08:53.943651706 +0000 UTC m=+1882.715944724"
Nov 03 20:08:54.934870 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:54.934825    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm" event={"ID":"fe2c855d-4f89-4546-b580-e5f7e22eb9bc","Type":"ContainerStarted","Data":"9535fd95d2bd30b629c9e2f4735293bbbb279ec784730475cbbe7b987f192942"}
Nov 03 20:08:54.935406 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:54.934920    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm"
Nov 03 20:08:54.954595 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:54.954540    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm" podStartSLOduration=2.954527884 podStartE2EDuration="2.954527884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:54.952819702 +0000 UTC m=+1883.725112720" watchObservedRunningTime="2026-11-03 20:08:54.954527884 +0000 UTC m=+1883.726820943"
Nov 03 20:08:56.759039 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:56.759006    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:08:58.282164 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:58.281789    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2" podUID="9f7d28df-3147-416e-aebb-d755624f00e4" containerName="main" containerID="cri-o://e23a86a66d007908f50798f95b0f0b95b42d1115b870bbcb74d5e083a80ca7ac" gracePeriod=45
Nov 03 20:08:58.774542 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:08:58.774431    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:08:58.870452 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:58.870421    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"
Nov 03 20:08:59.057281 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.056094    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-home\" (UniqueName: \"kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-home\") pod \"9f7d28df-3147-416e-aebb-d755624f00e4\" (UID: \"9f7d28df-3147-416e-aebb-d755624f00e4\") "
Nov 03 20:08:59.057281 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.056248    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-tmp-dir\") pod \"9f7d28df-3147-416e-aebb-d755624f00e4\" (UID: \"9f7d28df-3147-416e-aebb-d755624f00e4\") "
Nov 03 20:08:59.057281 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.056366    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f7d28df-3147-416e-aebb-d755624f00e4-kube-api-access-5zjnt\" (UniqueName: \"kubernetes.io/projected/9f7d28df-3147-416e-aebb-d755624f00e4-kube-api-access-5zjnt\") pod \"9f7d28df-3147-416e-aebb-d755624f00e4\" (UID: \"9f7d28df-3147-416e-aebb-d755624f00e4\") "
Nov 03 20:08:59.057281 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.056436    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-dshm\") pod \"9f7d28df-3147-416e-aebb-d755624f00e4\" (UID: \"9f7d28df-3147-416e-aebb-d755624f00e4\") "
Nov 03 20:08:59.057281 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.056483    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-model-cache\") pod \"9f7d28df-3147-416e-aebb-d755624f00e4\" (UID: \"9f7d28df-3147-416e-aebb-d755624f00e4\") "
Nov 03 20:08:59.057281 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.056531    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f7d28df-3147-416e-aebb-d755624f00e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f7d28df-3147-416e-aebb-d755624f00e4-tls-certs\") pod \"9f7d28df-3147-416e-aebb-d755624f00e4\" (UID: \"9f7d28df-3147-416e-aebb-d755624f00e4\") "
Nov 03 20:08:59.073301 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.071758    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-home" pod "9f7d28df-3147-416e-aebb-d755624f00e4" (UID: "9f7d28df-3147-416e-aebb-d755624f00e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:59.073666 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.071785    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-dshm" pod "9f7d28df-3147-416e-aebb-d755624f00e4" (UID: "9f7d28df-3147-416e-aebb-d755624f00e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:59.073666 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.072699    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-model-cache" pod "9f7d28df-3147-416e-aebb-d755624f00e4" (UID: "9f7d28df-3147-416e-aebb-d755624f00e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:59.073814 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.073604    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-tmp-dir" pod "9f7d28df-3147-416e-aebb-d755624f00e4" (UID: "9f7d28df-3147-416e-aebb-d755624f00e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:08:59.083667 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.083616    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7d28df-3147-416e-aebb-d755624f00e4-kube-api-access-5zjnt" pod "9f7d28df-3147-416e-aebb-d755624f00e4" (UID: "9f7d28df-3147-416e-aebb-d755624f00e4"). InnerVolumeSpecName "kube-api-access-5zjnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:08:59.086494 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.086463    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f7d28df-3147-416e-aebb-d755624f00e4-tls-certs" pod "9f7d28df-3147-416e-aebb-d755624f00e4" (UID: "9f7d28df-3147-416e-aebb-d755624f00e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:08:59.090140 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.090104    5591 generic.go:356] "Generic (PLEG): container finished" podID="9f7d28df-3147-416e-aebb-d755624f00e4" containerID="e23a86a66d007908f50798f95b0f0b95b42d1115b870bbcb74d5e083a80ca7ac" exitCode=0
Nov 03 20:08:59.090338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.090157    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2" event={"ID":"9f7d28df-3147-416e-aebb-d755624f00e4","Type":"ContainerDied","Data":"e23a86a66d007908f50798f95b0f0b95b42d1115b870bbcb74d5e083a80ca7ac"}
Nov 03 20:08:59.090338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.090171    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"
Nov 03 20:08:59.090338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.090190    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2" event={"ID":"9f7d28df-3147-416e-aebb-d755624f00e4","Type":"ContainerDied","Data":"47a486c4dbe42d56b14ed882527d03438e93241b74abe4f4be468d8fe1b804a3"}
Nov 03 20:08:59.090338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.090223    5591 scope.go:122] "RemoveContainer" containerID="e23a86a66d007908f50798f95b0f0b95b42d1115b870bbcb74d5e083a80ca7ac"
Nov 03 20:08:59.117253 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.117187    5591 scope.go:122] "RemoveContainer" containerID="e23a86a66d007908f50798f95b0f0b95b42d1115b870bbcb74d5e083a80ca7ac"
Nov 03 20:08:59.117755 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:08:59.117722    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e23a86a66d007908f50798f95b0f0b95b42d1115b870bbcb74d5e083a80ca7ac\": container with ID starting with e23a86a66d007908f50798f95b0f0b95b42d1115b870bbcb74d5e083a80ca7ac not found: ID does not exist" containerID="e23a86a66d007908f50798f95b0f0b95b42d1115b870bbcb74d5e083a80ca7ac"
Nov 03 20:08:59.117864 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.117771    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e23a86a66d007908f50798f95b0f0b95b42d1115b870bbcb74d5e083a80ca7ac"} err="failed to get container status \"e23a86a66d007908f50798f95b0f0b95b42d1115b870bbcb74d5e083a80ca7ac\": rpc error: code = NotFound desc = could not find container \"e23a86a66d007908f50798f95b0f0b95b42d1115b870bbcb74d5e083a80ca7ac\": container with ID starting with e23a86a66d007908f50798f95b0f0b95b42d1115b870bbcb74d5e083a80ca7ac not found: ID does not exist"
Nov 03 20:08:59.129687 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.129660    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"]
Nov 03 20:08:59.145484 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.145455    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2"]
Nov 03 20:08:59.161098 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.161063    5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-dshm\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:59.161098 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.161091    5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-model-cache\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:59.161098 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.161103    5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f7d28df-3147-416e-aebb-d755624f00e4-tls-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:59.161330 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.161113    5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-home\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:59.161330 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.161122    5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f7d28df-3147-416e-aebb-d755624f00e4-tmp-dir\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:59.161330 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.161130    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zjnt\" (UniqueName: \"kubernetes.io/projected/9f7d28df-3147-416e-aebb-d755624f00e4-kube-api-access-5zjnt\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:08:59.354300 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.354196    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f7d28df-3147-416e-aebb-d755624f00e4" path="/var/lib/kubelet/pods/9f7d28df-3147-416e-aebb-d755624f00e4/volumes"
Nov 03 20:08:59.775472 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:08:59.775409    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-bcfde8/auth-test-model-bcfde8-kserve-67794bd4cc-2fmq2" podUID="9f7d28df-3147-416e-aebb-d755624f00e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.186:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:09:00.404180 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.404149    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"]
Nov 03 20:09:00.405035 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.405018    5591 cpu_manager.go:414] "containerMap: removing container" podUID="9f7d28df-3147-416e-aebb-d755624f00e4" containerName="main"
Nov 03 20:09:00.405106 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.405036    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f7d28df-3147-416e-aebb-d755624f00e4" containerName="main"
Nov 03 20:09:00.405178 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.405169    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f7d28df-3147-416e-aebb-d755624f00e4" containerName="main"
Nov 03 20:09:00.409923 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.409904    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.416398 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.416376    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-43c674-openshift-default-dockercfg-slz8b\""
Nov 03 20:09:00.454678 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.454641    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"]
Nov 03 20:09:00.494838 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.494796    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-workload-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.495022 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.494874    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55d57\" (UniqueName: \"kubernetes.io/projected/90331769-8442-4145-978d-63f3e1820556-kube-api-access-55d57\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.495151 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.495045    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-istio-data\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.495151 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.495094    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-workload-certs\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.495347 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.495289    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-istio-envoy\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.495347 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.495322    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90331769-8442-4145-978d-63f3e1820556-istio-podinfo\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.495517 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.495351    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-credential-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.495517 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.495380    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90331769-8442-4145-978d-63f3e1820556-istiod-ca-cert\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.495693 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.495671    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90331769-8442-4145-978d-63f3e1820556-istio-token\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.600297 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.598699    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55d57\" (UniqueName: \"kubernetes.io/projected/90331769-8442-4145-978d-63f3e1820556-kube-api-access-55d57\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.600297 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.598857    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-istio-data\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.600297 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.598893    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-workload-certs\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.600297 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.599008    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-istio-envoy\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.600297 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.599037    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90331769-8442-4145-978d-63f3e1820556-istio-podinfo\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.600297 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.599061    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-credential-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.600297 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.599093    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90331769-8442-4145-978d-63f3e1820556-istiod-ca-cert\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.600297 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.599144    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90331769-8442-4145-978d-63f3e1820556-istio-token\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.600297 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.599331    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-workload-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.611243 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.610795    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-workload-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.611243 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.611175    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-istio-data\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.611553 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.611483    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-workload-certs\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.616798 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.615464    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-credential-socket\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.629796 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.629713    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90331769-8442-4145-978d-63f3e1820556-istiod-ca-cert\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.643961 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.643925    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55d57\" (UniqueName: \"kubernetes.io/projected/90331769-8442-4145-978d-63f3e1820556-kube-api-access-55d57\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.664385 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.663748    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90331769-8442-4145-978d-63f3e1820556-istio-token\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.664385 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.663959    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-istio-envoy\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.664385 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.664279    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90331769-8442-4145-978d-63f3e1820556-istio-podinfo\") pod \"e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") " pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:00.727505 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:00.727466    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:01.105954 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:09:01.105921    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90331769_8442_4145_978d_63f3e1820556.slice/crio-ec1df8c1008343113b808700836aba7ed4f2954385ee805497e4a83c2a277332 WatchSource:0}: Error finding container ec1df8c1008343113b808700836aba7ed4f2954385ee805497e4a83c2a277332: Status 404 returned error can't find the container with id ec1df8c1008343113b808700836aba7ed4f2954385ee805497e4a83c2a277332
Nov 03 20:09:01.106736 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:01.106712    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"]
Nov 03 20:09:01.109080 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:01.109035    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:09:01.109184 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:01.109128    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:09:01.109184 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:01.109168    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:09:01.442559 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:01.442439    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"]
Nov 03 20:09:01.443426 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:01.443363    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb" podUID="fa169e55-a457-4166-88ab-c65b6081d08e" containerName="istio-proxy" containerID="cri-o://933b9e58dc85e048a725a99718543b410c566438a639d79df807c099d4cea182" gracePeriod=30
Nov 03 20:09:02.122895 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:02.122859    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5" event={"ID":"90331769-8442-4145-978d-63f3e1820556","Type":"ContainerStarted","Data":"605da35f2c95889c5931dd414bcfc14a985d180434b35d35a2c32f919c44d781"}
Nov 03 20:09:02.123121 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:02.122915    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5" event={"ID":"90331769-8442-4145-978d-63f3e1820556","Type":"ContainerStarted","Data":"ec1df8c1008343113b808700836aba7ed4f2954385ee805497e4a83c2a277332"}
Nov 03 20:09:02.199265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:02.186529    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5" podStartSLOduration=2.186508247 podStartE2EDuration="2.186508247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:02.151938332 +0000 UTC m=+1890.924231353" watchObservedRunningTime="2026-11-03 20:09:02.186508247 +0000 UTC m=+1890.958801268"
Nov 03 20:09:02.735231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:02.734424    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:02.735231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:02.734558    5591 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Nov 03 20:09:02.735231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:02.734604    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5" podUID="90331769-8442-4145-978d-63f3e1820556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Nov 03 20:09:03.728637 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:03.728584    5591 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Nov 03 20:09:03.728861 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:03.728657    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5" podUID="90331769-8442-4145-978d-63f3e1820556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Nov 03 20:09:04.831194 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:04.826264    5591 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Nov 03 20:09:04.831194 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:04.826329    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5" podUID="90331769-8442-4145-978d-63f3e1820556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Nov 03 20:09:04.938375 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:04.938342    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27"
Nov 03 20:09:05.728323 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:05.728276    5591 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Nov 03 20:09:05.728552 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:05.728340    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5" podUID="90331769-8442-4145-978d-63f3e1820556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Nov 03 20:09:05.945837 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:05.945812    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm"
Nov 03 20:09:06.728144 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:06.728054    5591 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Nov 03 20:09:06.728144 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:06.728123    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5" podUID="90331769-8442-4145-978d-63f3e1820556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Nov 03 20:09:07.172584 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.172317    5591 generic.go:356] "Generic (PLEG): container finished" podID="fa169e55-a457-4166-88ab-c65b6081d08e" containerID="933b9e58dc85e048a725a99718543b410c566438a639d79df807c099d4cea182" exitCode=0
Nov 03 20:09:07.172584 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.172518    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb" event={"ID":"fa169e55-a457-4166-88ab-c65b6081d08e","Type":"ContainerDied","Data":"933b9e58dc85e048a725a99718543b410c566438a639d79df807c099d4cea182"}
Nov 03 20:09:07.308631 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.308556    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:09:07.457909 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.457873    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-istio-envoy\") pod \"fa169e55-a457-4166-88ab-c65b6081d08e\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") "
Nov 03 20:09:07.458127 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.457941    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa169e55-a457-4166-88ab-c65b6081d08e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa169e55-a457-4166-88ab-c65b6081d08e-istiod-ca-cert\") pod \"fa169e55-a457-4166-88ab-c65b6081d08e\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") "
Nov 03 20:09:07.458127 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.458001    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-credential-socket\") pod \"fa169e55-a457-4166-88ab-c65b6081d08e\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") "
Nov 03 20:09:07.458127 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.458104    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-workload-socket\") pod \"fa169e55-a457-4166-88ab-c65b6081d08e\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") "
Nov 03 20:09:07.458317 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.458258    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-istio-data\") pod \"fa169e55-a457-4166-88ab-c65b6081d08e\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") "
Nov 03 20:09:07.458317 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.458286    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa169e55-a457-4166-88ab-c65b6081d08e-kube-api-access-nt89x\" (UniqueName: \"kubernetes.io/projected/fa169e55-a457-4166-88ab-c65b6081d08e-kube-api-access-nt89x\") pod \"fa169e55-a457-4166-88ab-c65b6081d08e\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") "
Nov 03 20:09:07.458431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.458332    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa169e55-a457-4166-88ab-c65b6081d08e-istio-token\" (UniqueName: \"kubernetes.io/projected/fa169e55-a457-4166-88ab-c65b6081d08e-istio-token\") pod \"fa169e55-a457-4166-88ab-c65b6081d08e\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") "
Nov 03 20:09:07.458431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.458330    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-credential-socket" pod "fa169e55-a457-4166-88ab-c65b6081d08e" (UID: "fa169e55-a457-4166-88ab-c65b6081d08e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:07.458431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.458356    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa169e55-a457-4166-88ab-c65b6081d08e-istiod-ca-cert" pod "fa169e55-a457-4166-88ab-c65b6081d08e" (UID: "fa169e55-a457-4166-88ab-c65b6081d08e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:09:07.458431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.458376    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-workload-certs\") pod \"fa169e55-a457-4166-88ab-c65b6081d08e\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") "
Nov 03 20:09:07.458635 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.458429    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa169e55-a457-4166-88ab-c65b6081d08e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa169e55-a457-4166-88ab-c65b6081d08e-istio-podinfo\") pod \"fa169e55-a457-4166-88ab-c65b6081d08e\" (UID: \"fa169e55-a457-4166-88ab-c65b6081d08e\") "
Nov 03 20:09:07.458635 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.458540    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-istio-data" pod "fa169e55-a457-4166-88ab-c65b6081d08e" (UID: "fa169e55-a457-4166-88ab-c65b6081d08e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:07.458635 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.458557    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-workload-socket" pod "fa169e55-a457-4166-88ab-c65b6081d08e" (UID: "fa169e55-a457-4166-88ab-c65b6081d08e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:07.458801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.458756    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-workload-certs" pod "fa169e55-a457-4166-88ab-c65b6081d08e" (UID: "fa169e55-a457-4166-88ab-c65b6081d08e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:07.459218 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.459104    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa169e55-a457-4166-88ab-c65b6081d08e-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:07.459218 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.459129    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:07.459218 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.459143    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:07.459218 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.459157    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:07.459218 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.459171    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:07.460941 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.460907    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-istio-envoy" pod "fa169e55-a457-4166-88ab-c65b6081d08e" (UID: "fa169e55-a457-4166-88ab-c65b6081d08e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:07.461433 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.461409    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa169e55-a457-4166-88ab-c65b6081d08e-istio-token" pod "fa169e55-a457-4166-88ab-c65b6081d08e" (UID: "fa169e55-a457-4166-88ab-c65b6081d08e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:09:07.461528 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.461515    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa169e55-a457-4166-88ab-c65b6081d08e-kube-api-access-nt89x" pod "fa169e55-a457-4166-88ab-c65b6081d08e" (UID: "fa169e55-a457-4166-88ab-c65b6081d08e"). InnerVolumeSpecName "kube-api-access-nt89x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:09:07.461582 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.461557    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa169e55-a457-4166-88ab-c65b6081d08e-istio-podinfo" pod "fa169e55-a457-4166-88ab-c65b6081d08e" (UID: "fa169e55-a457-4166-88ab-c65b6081d08e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:09:07.560758 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.560671    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt89x\" (UniqueName: \"kubernetes.io/projected/fa169e55-a457-4166-88ab-c65b6081d08e-kube-api-access-nt89x\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:07.560758 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.560704    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa169e55-a457-4166-88ab-c65b6081d08e-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:07.560758 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.560714    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa169e55-a457-4166-88ab-c65b6081d08e-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:07.560758 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.560723    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa169e55-a457-4166-88ab-c65b6081d08e-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:07.728042 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.728002    5591 patch_prober.go:29] interesting pod/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Nov 03 20:09:07.728309 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:07.728062    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5" podUID="90331769-8442-4145-978d-63f3e1820556" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Nov 03 20:09:08.183075 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.183045    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"
Nov 03 20:09:08.183653 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.183078    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb" event={"ID":"fa169e55-a457-4166-88ab-c65b6081d08e","Type":"ContainerDied","Data":"a67fd9a03d27d0403fd76fcb693b1cd6d4dc3cd8670f06a8b40da426f03af23a"}
Nov 03 20:09:08.183653 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.183187    5591 scope.go:122] "RemoveContainer" containerID="933b9e58dc85e048a725a99718543b410c566438a639d79df807c099d4cea182"
Nov 03 20:09:08.211046 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.211019    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"]
Nov 03 20:09:08.218822 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.218791    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-xk5kb"]
Nov 03 20:09:08.565141 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.565108    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4"]
Nov 03 20:09:08.566192 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.566156    5591 cpu_manager.go:414] "containerMap: removing container" podUID="fa169e55-a457-4166-88ab-c65b6081d08e" containerName="istio-proxy"
Nov 03 20:09:08.566192 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.566177    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa169e55-a457-4166-88ab-c65b6081d08e" containerName="istio-proxy"
Nov 03 20:09:08.566434 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.566343    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa169e55-a457-4166-88ab-c65b6081d08e" containerName="istio-proxy"
Nov 03 20:09:08.571974 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.571932    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4"
Nov 03 20:09:08.583192 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.583169    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-8b7351-serving-cert\""
Nov 03 20:09:08.584357 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.584324    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb496cd-b12d-4331-af8b-417c6fbffdf2-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4\" (UID: \"fbb496cd-b12d-4331-af8b-417c6fbffdf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4"
Nov 03 20:09:08.584522 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.584502    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbb496cd-b12d-4331-af8b-417c6fbffdf2-metrics-certs\") pod \"maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4\" (UID: \"fbb496cd-b12d-4331-af8b-417c6fbffdf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4"
Nov 03 20:09:08.584758 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.584735    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szxsb\" (UniqueName: \"kubernetes.io/projected/fbb496cd-b12d-4331-af8b-417c6fbffdf2-kube-api-access-szxsb\") pod \"maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4\" (UID: \"fbb496cd-b12d-4331-af8b-417c6fbffdf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4"
Nov 03 20:09:08.584900 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.584877    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb496cd-b12d-4331-af8b-417c6fbffdf2-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4\" (UID: \"fbb496cd-b12d-4331-af8b-417c6fbffdf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4"
Nov 03 20:09:08.585515 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.585483    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4"]
Nov 03 20:09:08.685780 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.685742    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbb496cd-b12d-4331-af8b-417c6fbffdf2-metrics-certs\") pod \"maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4\" (UID: \"fbb496cd-b12d-4331-af8b-417c6fbffdf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4"
Nov 03 20:09:08.685943 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.685861    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szxsb\" (UniqueName: \"kubernetes.io/projected/fbb496cd-b12d-4331-af8b-417c6fbffdf2-kube-api-access-szxsb\") pod \"maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4\" (UID: \"fbb496cd-b12d-4331-af8b-417c6fbffdf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4"
Nov 03 20:09:08.686003 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.685973    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb496cd-b12d-4331-af8b-417c6fbffdf2-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4\" (UID: \"fbb496cd-b12d-4331-af8b-417c6fbffdf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4"
Nov 03 20:09:08.686037 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.686011    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb496cd-b12d-4331-af8b-417c6fbffdf2-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4\" (UID: \"fbb496cd-b12d-4331-af8b-417c6fbffdf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4"
Nov 03 20:09:08.686626 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.686600    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb496cd-b12d-4331-af8b-417c6fbffdf2-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4\" (UID: \"fbb496cd-b12d-4331-af8b-417c6fbffdf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4"
Nov 03 20:09:08.688476 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.688449    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbb496cd-b12d-4331-af8b-417c6fbffdf2-metrics-certs\") pod \"maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4\" (UID: \"fbb496cd-b12d-4331-af8b-417c6fbffdf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4"
Nov 03 20:09:08.688565 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.688512    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb496cd-b12d-4331-af8b-417c6fbffdf2-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4\" (UID: \"fbb496cd-b12d-4331-af8b-417c6fbffdf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4"
Nov 03 20:09:08.706510 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.706476    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szxsb\" (UniqueName: \"kubernetes.io/projected/fbb496cd-b12d-4331-af8b-417c6fbffdf2-kube-api-access-szxsb\") pod \"maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4\" (UID: \"fbb496cd-b12d-4331-af8b-417c6fbffdf2\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4"
Nov 03 20:09:08.753536 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.749576    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:08.761231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.756594    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:08.777228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.774537    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:09:08.902046 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:08.901960    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4"
Nov 03 20:09:09.304516 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:09:09.301597    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb496cd_b12d_4331_af8b_417c6fbffdf2.slice/crio-773f26cc0bb4657b2ecde6a5339b7004827cbbbbadc69d84641cca2c0b0fa1f1 WatchSource:0}: Error finding container 773f26cc0bb4657b2ecde6a5339b7004827cbbbbadc69d84641cca2c0b0fa1f1: Status 404 returned error can't find the container with id 773f26cc0bb4657b2ecde6a5339b7004827cbbbbadc69d84641cca2c0b0fa1f1
Nov 03 20:09:09.307506 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:09.306177    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4"]
Nov 03 20:09:09.368098 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:09.368051    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa169e55-a457-4166-88ab-c65b6081d08e" path="/var/lib/kubelet/pods/fa169e55-a457-4166-88ab-c65b6081d08e/volumes"
Nov 03 20:09:10.327025 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:10.326982    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4" event={"ID":"fbb496cd-b12d-4331-af8b-417c6fbffdf2","Type":"ContainerStarted","Data":"c236dd44ac0c3e5f98212c5d973bb16a0285ca834b2df1294d7e513583fad17a"}
Nov 03 20:09:10.327557 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:10.327061    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4" event={"ID":"fbb496cd-b12d-4331-af8b-417c6fbffdf2","Type":"ContainerStarted","Data":"773f26cc0bb4657b2ecde6a5339b7004827cbbbbadc69d84641cca2c0b0fa1f1"}
Nov 03 20:09:10.327557 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:10.327102    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4"
Nov 03 20:09:10.358349 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:10.358274    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4" podStartSLOduration=1.8304277519999999 podStartE2EDuration="2.358253386s" totalImagesPullingTime="527.825634ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:10.353606061 +0000 UTC m=+1899.125899099" watchObservedRunningTime="2026-11-03 20:09:10.358253386 +0000 UTC m=+1899.130546406"
Nov 03 20:09:16.546657 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:16.539126    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"]
Nov 03 20:09:16.548141 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:16.547403    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2" podUID="4b5fe641-913e-4f0d-81b6-1b6e2970e2e7" containerName="istio-proxy" containerID="cri-o://e047bc30be3d8c7dfdd94fdf0ed97f95a59fd061b9a20049a147f2b2e773476e" gracePeriod=30
Nov 03 20:09:17.388780 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:17.388748    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4"
Nov 03 20:09:18.427079 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:18.427039    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k"]
Nov 03 20:09:18.432327 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:18.432302    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k"
Nov 03 20:09:18.442451 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:18.442424    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-36556d\""
Nov 03 20:09:18.442645 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:18.442569    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-36556d-dockercfg-gkcxm\""
Nov 03 20:09:18.444380 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:18.444352    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k"]
Nov 03 20:09:18.499412 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:18.499372    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-36556d-f66cd46b-fz9td"]
Nov 03 20:09:18.513767 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:18.513736    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-36556d-f66cd46b-fz9td"]
Nov 03 20:09:18.513946 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:18.513870    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-36556d-f66cd46b-fz9td"
Nov 03 20:09:18.538673 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:18.538638    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76d28c6-0efd-4c21-bcf8-ed01189327b0-plugins-config-volume\") pod \"payload-processing-e2e-autores-36556d-f66cd46b-fz9td\" (UID: \"c76d28c6-0efd-4c21-bcf8-ed01189327b0\") " pod="openshift-ingress/payload-processing-e2e-autores-36556d-f66cd46b-fz9td"
Nov 03 20:09:18.538884 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:18.538862    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad53b126-3346-493f-bf9e-139e08b50cf5-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k\" (UID: \"ad53b126-3346-493f-bf9e-139e08b50cf5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k"
Nov 03 20:09:18.539019 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:18.539000    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrkhv\" (UniqueName: \"kubernetes.io/projected/c76d28c6-0efd-4c21-bcf8-ed01189327b0-kube-api-access-rrkhv\") pod \"payload-processing-e2e-autores-36556d-f66cd46b-fz9td\" (UID: \"c76d28c6-0efd-4c21-bcf8-ed01189327b0\") " pod="openshift-ingress/payload-processing-e2e-autores-36556d-f66cd46b-fz9td"
Nov 03 20:09:18.539087 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:18.539039    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6mlb\" (UniqueName: \"kubernetes.io/projected/ad53b126-3346-493f-bf9e-139e08b50cf5-kube-api-access-z6mlb\") pod \"payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k\" (UID: \"ad53b126-3346-493f-bf9e-139e08b50cf5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k"
Nov 03 20:09:18.714683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:18.714595    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad53b126-3346-493f-bf9e-139e08b50cf5-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k\" (UID: \"ad53b126-3346-493f-bf9e-139e08b50cf5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k"
Nov 03 20:09:18.715398 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:18.715376    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrkhv\" (UniqueName: \"kubernetes.io/projected/c76d28c6-0efd-4c21-bcf8-ed01189327b0-kube-api-access-rrkhv\") pod \"payload-processing-e2e-autores-36556d-f66cd46b-fz9td\" (UID: \"c76d28c6-0efd-4c21-bcf8-ed01189327b0\") " pod="openshift-ingress/payload-processing-e2e-autores-36556d-f66cd46b-fz9td"
Nov 03 20:09:18.715919 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:18.715899    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6mlb\" (UniqueName: \"kubernetes.io/projected/ad53b126-3346-493f-bf9e-139e08b50cf5-kube-api-access-z6mlb\") pod \"payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k\" (UID: \"ad53b126-3346-493f-bf9e-139e08b50cf5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k"
Nov 03 20:09:18.716467 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:18.716449    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76d28c6-0efd-4c21-bcf8-ed01189327b0-plugins-config-volume\") pod \"payload-processing-e2e-autores-36556d-f66cd46b-fz9td\" (UID: \"c76d28c6-0efd-4c21-bcf8-ed01189327b0\") " pod="openshift-ingress/payload-processing-e2e-autores-36556d-f66cd46b-fz9td"
Nov 03 20:09:18.718189 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:18.717944    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76d28c6-0efd-4c21-bcf8-ed01189327b0-plugins-config-volume\") pod \"payload-processing-e2e-autores-36556d-f66cd46b-fz9td\" (UID: \"c76d28c6-0efd-4c21-bcf8-ed01189327b0\") " pod="openshift-ingress/payload-processing-e2e-autores-36556d-f66cd46b-fz9td"
Nov 03 20:09:18.718640 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:18.718617    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad53b126-3346-493f-bf9e-139e08b50cf5-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k\" (UID: \"ad53b126-3346-493f-bf9e-139e08b50cf5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k"
Nov 03 20:09:18.776235 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:18.758616    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrkhv\" (UniqueName: \"kubernetes.io/projected/c76d28c6-0efd-4c21-bcf8-ed01189327b0-kube-api-access-rrkhv\") pod \"payload-processing-e2e-autores-36556d-f66cd46b-fz9td\" (UID: \"c76d28c6-0efd-4c21-bcf8-ed01189327b0\") " pod="openshift-ingress/payload-processing-e2e-autores-36556d-f66cd46b-fz9td"
Nov 03 20:09:18.777339 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:18.777305    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6mlb\" (UniqueName: \"kubernetes.io/projected/ad53b126-3346-493f-bf9e-139e08b50cf5-kube-api-access-z6mlb\") pod \"payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k\" (UID: \"ad53b126-3346-493f-bf9e-139e08b50cf5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k"
Nov 03 20:09:18.839253 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:18.838510    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-36556d-f66cd46b-fz9td"
Nov 03 20:09:19.094566 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:19.092063    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k"
Nov 03 20:09:19.199285 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:09:19.198161    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76d28c6_0efd_4c21_bcf8_ed01189327b0.slice/crio-d52b5d7fdd2988f2b72504011792b41351252ee0545890b6ef333ef48d11a85e WatchSource:0}: Error finding container d52b5d7fdd2988f2b72504011792b41351252ee0545890b6ef333ef48d11a85e: Status 404 returned error can't find the container with id d52b5d7fdd2988f2b72504011792b41351252ee0545890b6ef333ef48d11a85e
Nov 03 20:09:19.199285 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:19.198634    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-36556d-f66cd46b-fz9td"]
Nov 03 20:09:19.399903 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:19.399868    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k"]
Nov 03 20:09:19.497650 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:19.497606    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k" event={"ID":"ad53b126-3346-493f-bf9e-139e08b50cf5","Type":"ContainerStarted","Data":"564105209f811ee847cf6ef715c6763d4aeecbbd015f4f4bd357e0c60333ca7a"}
Nov 03 20:09:19.498104 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:19.497697    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k"
Nov 03 20:09:19.500651 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:19.500548    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-36556d-f66cd46b-fz9td" event={"ID":"c76d28c6-0efd-4c21-bcf8-ed01189327b0","Type":"ContainerStarted","Data":"1a0d5b107d312836e45c7e57dbb5f7a3047c1c1fbbf7c72994cbd0831792f3f7"}
Nov 03 20:09:19.500651 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:19.500613    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-36556d-f66cd46b-fz9td" event={"ID":"c76d28c6-0efd-4c21-bcf8-ed01189327b0","Type":"ContainerStarted","Data":"d52b5d7fdd2988f2b72504011792b41351252ee0545890b6ef333ef48d11a85e"}
Nov 03 20:09:19.500917 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:19.500785    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-36556d-f66cd46b-fz9td"
Nov 03 20:09:19.521552 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:19.521448    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k" podStartSLOduration=1.521434426 podStartE2EDuration="1.521434426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:19.516856649 +0000 UTC m=+1908.289149669" watchObservedRunningTime="2026-11-03 20:09:19.521434426 +0000 UTC m=+1908.293727445"
Nov 03 20:09:19.535123 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:19.535066    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-36556d-f66cd46b-fz9td" podStartSLOduration=1.535051132 podStartE2EDuration="1.535051132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:19.534681853 +0000 UTC m=+1908.306974873" watchObservedRunningTime="2026-11-03 20:09:19.535051132 +0000 UTC m=+1908.307344149"
Nov 03 20:09:20.512731 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:20.512690    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k" event={"ID":"ad53b126-3346-493f-bf9e-139e08b50cf5","Type":"ContainerStarted","Data":"197468be71b58f838fae5ea39a3cffe8d90a90619fc5175bcdbc1aec4180d143"}
Nov 03 20:09:22.082081 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.082047    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 20:09:22.084186 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.084160    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-token\" (UniqueName: \"kubernetes.io/projected/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-token\") pod \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") "
Nov 03 20:09:22.084343 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.084252    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-workload-certs\") pod \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") "
Nov 03 20:09:22.084343 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.084283    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-credential-socket\") pod \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") "
Nov 03 20:09:22.084487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.084372    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-podinfo\") pod \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") "
Nov 03 20:09:22.084487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.084425    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-envoy\") pod \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") "
Nov 03 20:09:22.084623 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.084530    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-data\") pod \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") "
Nov 03 20:09:22.084623 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.084524    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-workload-certs" pod "4b5fe641-913e-4f0d-81b6-1b6e2970e2e7" (UID: "4b5fe641-913e-4f0d-81b6-1b6e2970e2e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:22.084623 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.084589    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-credential-socket" pod "4b5fe641-913e-4f0d-81b6-1b6e2970e2e7" (UID: "4b5fe641-913e-4f0d-81b6-1b6e2970e2e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:22.084623 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.084601    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-workload-socket\") pod \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") "
Nov 03 20:09:22.084823 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.084629    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-kube-api-access-zdflb\" (UniqueName: \"kubernetes.io/projected/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-kube-api-access-zdflb\") pod \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") "
Nov 03 20:09:22.084823 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.084688    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istiod-ca-cert\") pod \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\" (UID: \"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7\") "
Nov 03 20:09:22.084823 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.084812    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-workload-socket" pod "4b5fe641-913e-4f0d-81b6-1b6e2970e2e7" (UID: "4b5fe641-913e-4f0d-81b6-1b6e2970e2e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:22.084989 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.084827    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-data" pod "4b5fe641-913e-4f0d-81b6-1b6e2970e2e7" (UID: "4b5fe641-913e-4f0d-81b6-1b6e2970e2e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:22.085080 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.085061    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istiod-ca-cert" pod "4b5fe641-913e-4f0d-81b6-1b6e2970e2e7" (UID: "4b5fe641-913e-4f0d-81b6-1b6e2970e2e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:09:22.085341 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.085230    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:22.085341 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.085244    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:22.085341 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.085253    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:22.085341 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.085261    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:22.085341 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.085269    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:22.086760 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.086729    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-token" pod "4b5fe641-913e-4f0d-81b6-1b6e2970e2e7" (UID: "4b5fe641-913e-4f0d-81b6-1b6e2970e2e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:09:22.087014 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.086994    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-podinfo" pod "4b5fe641-913e-4f0d-81b6-1b6e2970e2e7" (UID: "4b5fe641-913e-4f0d-81b6-1b6e2970e2e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:09:22.087163 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.087147    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-envoy" pod "4b5fe641-913e-4f0d-81b6-1b6e2970e2e7" (UID: "4b5fe641-913e-4f0d-81b6-1b6e2970e2e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:09:22.087333 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.087308    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-kube-api-access-zdflb" pod "4b5fe641-913e-4f0d-81b6-1b6e2970e2e7" (UID: "4b5fe641-913e-4f0d-81b6-1b6e2970e2e7"). InnerVolumeSpecName "kube-api-access-zdflb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:09:22.186315 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.186281    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:22.186315 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.186312    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:22.186315 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.186322    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:22.186552 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.186331    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdflb\" (UniqueName: \"kubernetes.io/projected/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7-kube-api-access-zdflb\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:09:22.538442 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.538404    5591 generic.go:356] "Generic (PLEG): container finished" podID="4b5fe641-913e-4f0d-81b6-1b6e2970e2e7" containerID="e047bc30be3d8c7dfdd94fdf0ed97f95a59fd061b9a20049a147f2b2e773476e" exitCode=0
Nov 03 20:09:22.538633 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.538457    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"
Nov 03 20:09:22.538633 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.538499    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2" event={"ID":"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7","Type":"ContainerDied","Data":"e047bc30be3d8c7dfdd94fdf0ed97f95a59fd061b9a20049a147f2b2e773476e"}
Nov 03 20:09:22.538633 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.538552    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2" event={"ID":"4b5fe641-913e-4f0d-81b6-1b6e2970e2e7","Type":"ContainerDied","Data":"b3105b478cf529d71f7ca943a1c3a66554f87a2da37cc5bead946961da445795"}
Nov 03 20:09:22.538633 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.538573    5591 scope.go:122] "RemoveContainer" containerID="e047bc30be3d8c7dfdd94fdf0ed97f95a59fd061b9a20049a147f2b2e773476e"
Nov 03 20:09:22.551954 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.551935    5591 scope.go:122] "RemoveContainer" containerID="e047bc30be3d8c7dfdd94fdf0ed97f95a59fd061b9a20049a147f2b2e773476e"
Nov 03 20:09:22.552303 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:09:22.552276    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e047bc30be3d8c7dfdd94fdf0ed97f95a59fd061b9a20049a147f2b2e773476e\": container with ID starting with e047bc30be3d8c7dfdd94fdf0ed97f95a59fd061b9a20049a147f2b2e773476e not found: ID does not exist" containerID="e047bc30be3d8c7dfdd94fdf0ed97f95a59fd061b9a20049a147f2b2e773476e"
Nov 03 20:09:22.552397 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.552315    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e047bc30be3d8c7dfdd94fdf0ed97f95a59fd061b9a20049a147f2b2e773476e"} err="failed to get container status \"e047bc30be3d8c7dfdd94fdf0ed97f95a59fd061b9a20049a147f2b2e773476e\": rpc error: code = NotFound desc = could not find container \"e047bc30be3d8c7dfdd94fdf0ed97f95a59fd061b9a20049a147f2b2e773476e\": container with ID starting with e047bc30be3d8c7dfdd94fdf0ed97f95a59fd061b9a20049a147f2b2e773476e not found: ID does not exist"
Nov 03 20:09:22.595548 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.595487    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"]
Nov 03 20:09:22.640792 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:22.640719    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-n7pn2"]
Nov 03 20:09:23.398237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:23.392648    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b5fe641-913e-4f0d-81b6-1b6e2970e2e7" path="/var/lib/kubelet/pods/4b5fe641-913e-4f0d-81b6-1b6e2970e2e7/volumes"
Nov 03 20:09:28.789228 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:09:28.783917    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:09:30.518663 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:30.518633    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-36556d-f66cd46b-fz9td"
Nov 03 20:09:31.532509 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:31.532481    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k"
Nov 03 20:09:35.458117 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:35.458078    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7"]
Nov 03 20:09:35.459494 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:35.459471    5591 cpu_manager.go:414] "containerMap: removing container" podUID="4b5fe641-913e-4f0d-81b6-1b6e2970e2e7" containerName="istio-proxy"
Nov 03 20:09:35.459557 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:35.459498    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b5fe641-913e-4f0d-81b6-1b6e2970e2e7" containerName="istio-proxy"
Nov 03 20:09:35.459722 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:35.459709    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b5fe641-913e-4f0d-81b6-1b6e2970e2e7" containerName="istio-proxy"
Nov 03 20:09:35.465966 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:35.465942    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7"
Nov 03 20:09:35.479032 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:35.475176    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-36556d-serving-cert\""
Nov 03 20:09:35.481185 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:35.481155    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7"]
Nov 03 20:09:35.546353 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:35.546270    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1f58d2-241a-45ee-8e9c-32def711457b-metrics-certs\") pod \"maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7\" (UID: \"3b1f58d2-241a-45ee-8e9c-32def711457b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7"
Nov 03 20:09:35.546729 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:35.546490    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1f58d2-241a-45ee-8e9c-32def711457b-openshift-service-ca\") pod \"maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7\" (UID: \"3b1f58d2-241a-45ee-8e9c-32def711457b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7"
Nov 03 20:09:35.546729 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:35.546626    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdbnw\" (UniqueName: \"kubernetes.io/projected/3b1f58d2-241a-45ee-8e9c-32def711457b-kube-api-access-tdbnw\") pod \"maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7\" (UID: \"3b1f58d2-241a-45ee-8e9c-32def711457b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7"
Nov 03 20:09:35.546729 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:35.546664    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b1f58d2-241a-45ee-8e9c-32def711457b-maas-api-tls\") pod \"maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7\" (UID: \"3b1f58d2-241a-45ee-8e9c-32def711457b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7"
Nov 03 20:09:35.648607 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:35.648574    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1f58d2-241a-45ee-8e9c-32def711457b-openshift-service-ca\") pod \"maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7\" (UID: \"3b1f58d2-241a-45ee-8e9c-32def711457b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7"
Nov 03 20:09:35.648786 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:35.648670    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdbnw\" (UniqueName: \"kubernetes.io/projected/3b1f58d2-241a-45ee-8e9c-32def711457b-kube-api-access-tdbnw\") pod \"maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7\" (UID: \"3b1f58d2-241a-45ee-8e9c-32def711457b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7"
Nov 03 20:09:35.648786 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:35.648692    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b1f58d2-241a-45ee-8e9c-32def711457b-maas-api-tls\") pod \"maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7\" (UID: \"3b1f58d2-241a-45ee-8e9c-32def711457b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7"
Nov 03 20:09:35.648905 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:35.648802    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1f58d2-241a-45ee-8e9c-32def711457b-metrics-certs\") pod \"maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7\" (UID: \"3b1f58d2-241a-45ee-8e9c-32def711457b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7"
Nov 03 20:09:35.649427 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:35.649386    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1f58d2-241a-45ee-8e9c-32def711457b-openshift-service-ca\") pod \"maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7\" (UID: \"3b1f58d2-241a-45ee-8e9c-32def711457b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7"
Nov 03 20:09:35.651709 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:35.651679    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1f58d2-241a-45ee-8e9c-32def711457b-metrics-certs\") pod \"maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7\" (UID: \"3b1f58d2-241a-45ee-8e9c-32def711457b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7"
Nov 03 20:09:35.651839 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:35.651679    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b1f58d2-241a-45ee-8e9c-32def711457b-maas-api-tls\") pod \"maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7\" (UID: \"3b1f58d2-241a-45ee-8e9c-32def711457b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7"
Nov 03 20:09:35.695249 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:35.661772    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdbnw\" (UniqueName: \"kubernetes.io/projected/3b1f58d2-241a-45ee-8e9c-32def711457b-kube-api-access-tdbnw\") pod \"maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7\" (UID: \"3b1f58d2-241a-45ee-8e9c-32def711457b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7"
Nov 03 20:09:35.799240 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:35.795917    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7"
Nov 03 20:09:36.232380 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:36.232345    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7"]
Nov 03 20:09:36.235354 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:09:36.235323    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1f58d2_241a_45ee_8e9c_32def711457b.slice/crio-c47651426b8ccd18d7939deb417365e3bdc0087bbec3b6703f00788e353556ad WatchSource:0}: Error finding container c47651426b8ccd18d7939deb417365e3bdc0087bbec3b6703f00788e353556ad: Status 404 returned error can't find the container with id c47651426b8ccd18d7939deb417365e3bdc0087bbec3b6703f00788e353556ad
Nov 03 20:09:36.880951 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:36.880913    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7" event={"ID":"3b1f58d2-241a-45ee-8e9c-32def711457b","Type":"ContainerStarted","Data":"d532ac953c44a308002a487e9dda01bcf3bf668a9e528ddc606c061ba639b8d3"}
Nov 03 20:09:36.881436 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:36.880968    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7" event={"ID":"3b1f58d2-241a-45ee-8e9c-32def711457b","Type":"ContainerStarted","Data":"c47651426b8ccd18d7939deb417365e3bdc0087bbec3b6703f00788e353556ad"}
Nov 03 20:09:36.881436 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:36.881055    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7"
Nov 03 20:09:36.903857 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:36.903792    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7" podStartSLOduration=1.408430951 podStartE2EDuration="1.903777998s" totalImagesPullingTime="495.347047ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:09:36.901847431 +0000 UTC m=+1925.674140449" watchObservedRunningTime="2026-11-03 20:09:36.903777998 +0000 UTC m=+1925.676071053"
Nov 03 20:09:42.895645 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:42.895607    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7"
Nov 03 20:09:50.263483 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:50.263451    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:09:58.727824 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:09:58.727786    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:09:58.827800 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:58.827765    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"]
Nov 03 20:09:58.832559 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:58.832538    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:58.845680 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:58.845652    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"]
Nov 03 20:09:58.969438 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:58.969404    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-f154ac-openshift-default-dockercfg-xndgl\""
Nov 03 20:09:59.030326 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.018759    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6k2k\" (UniqueName: \"kubernetes.io/projected/146cc739-235f-46e7-a767-1734c9127951-kube-api-access-s6k2k\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.030326 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.018897    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-istio-envoy\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.030326 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.018947    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-workload-certs\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.030326 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.019069    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-workload-socket\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.030326 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.019185    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146cc739-235f-46e7-a767-1734c9127951-istiod-ca-cert\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.030326 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.020178    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-credential-socket\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.030326 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.020255    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-istio-data\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.030326 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.020396    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146cc739-235f-46e7-a767-1734c9127951-istio-podinfo\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.030326 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.020443    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/146cc739-235f-46e7-a767-1734c9127951-istio-token\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.122025 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.121985    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-istio-envoy\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.122263 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.122040    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-workload-certs\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.122263 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.122162    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-workload-socket\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.122263 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.122257    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146cc739-235f-46e7-a767-1734c9127951-istiod-ca-cert\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.122445 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.122312    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-credential-socket\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.122445 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.122355    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-istio-data\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.122445 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.122416    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-workload-certs\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.122589 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.122497    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146cc739-235f-46e7-a767-1734c9127951-istio-podinfo\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.122589 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.122524    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/146cc739-235f-46e7-a767-1734c9127951-istio-token\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.122589 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.122578    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6k2k\" (UniqueName: \"kubernetes.io/projected/146cc739-235f-46e7-a767-1734c9127951-kube-api-access-s6k2k\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.123013 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.122987    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146cc739-235f-46e7-a767-1734c9127951-istiod-ca-cert\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.123313 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.123284    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-workload-socket\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.123487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.123467    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-credential-socket\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.123535 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.123480    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-istio-data\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.125910 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.125883    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-istio-envoy\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.126116 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.126099    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146cc739-235f-46e7-a767-1734c9127951-istio-podinfo\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.135106 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.135080    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/146cc739-235f-46e7-a767-1734c9127951-istio-token\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.135250 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.135088    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6k2k\" (UniqueName: \"kubernetes.io/projected/146cc739-235f-46e7-a767-1734c9127951-kube-api-access-s6k2k\") pod \"e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") " pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.150199 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.150172    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:09:59.442330 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.442301    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"]
Nov 03 20:09:59.443523 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:09:59.443499    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146cc739_235f_46e7_a767_1734c9127951.slice/crio-9a0294b249a094e710039c55d8364ff1498793d8689b5cd2e33dfc4746d7ae8a WatchSource:0}: Error finding container 9a0294b249a094e710039c55d8364ff1498793d8689b5cd2e33dfc4746d7ae8a: Status 404 returned error can't find the container with id 9a0294b249a094e710039c55d8364ff1498793d8689b5cd2e33dfc4746d7ae8a
Nov 03 20:09:59.446848 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.446813    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:09:59.446935 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.446899    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:09:59.446935 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:09:59.446929    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:10:00.170691 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:00.170654    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68" event={"ID":"146cc739-235f-46e7-a767-1734c9127951","Type":"ContainerStarted","Data":"f1a723a7fa352d67f613b92b63e66f01243a440ad53dc6019a675f6d47638b5e"}
Nov 03 20:10:00.171114 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:00.170718    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68" event={"ID":"146cc739-235f-46e7-a767-1734c9127951","Type":"ContainerStarted","Data":"9a0294b249a094e710039c55d8364ff1498793d8689b5cd2e33dfc4746d7ae8a"}
Nov 03 20:10:00.197067 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:00.197009    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68" podStartSLOduration=2.196994522 podStartE2EDuration="2.196994522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:00.195663703 +0000 UTC m=+1948.967956722" watchObservedRunningTime="2026-11-03 20:10:00.196994522 +0000 UTC m=+1948.969287539"
Nov 03 20:10:01.151271 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:01.151238    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:10:01.152486 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:01.152461    5591 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 03 20:10:01.152579 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:01.152520    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68" podUID="146cc739-235f-46e7-a767-1734c9127951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 03 20:10:02.150673 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:02.150634    5591 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 03 20:10:02.151067 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:02.150713    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68" podUID="146cc739-235f-46e7-a767-1734c9127951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 03 20:10:03.150736 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:03.150695    5591 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 03 20:10:03.151455 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:03.151412    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68" podUID="146cc739-235f-46e7-a767-1734c9127951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 03 20:10:04.151230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:04.150642    5591 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 03 20:10:04.151230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:04.150728    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68" podUID="146cc739-235f-46e7-a767-1734c9127951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 03 20:10:05.151263 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:05.151218    5591 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 03 20:10:05.151794 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:05.151303    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68" podUID="146cc739-235f-46e7-a767-1734c9127951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 03 20:10:06.151407 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:06.151370    5591 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body=
Nov 03 20:10:06.151822 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:06.151439    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68" podUID="146cc739-235f-46e7-a767-1734c9127951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused"
Nov 03 20:10:07.155729 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:07.155698    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:10:07.156196 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:07.156107    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:10:07.156929 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:07.156908    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:10:12.753967 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:12.753929    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"]
Nov 03 20:10:12.761571 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:12.761546    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:12.781009 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:12.780528    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"]
Nov 03 20:10:12.854767 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:12.854738    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-36556d-openshift-default-dockercfg-hnpg9\""
Nov 03 20:10:12.924763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:12.924733    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/740ca9b8-631f-4381-b864-22b201e29d11-istio-token\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-5n8rn\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:12.924948 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:12.924849    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-istio-data\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-5n8rn\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:12.924948 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:12.924891    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-workload-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-5n8rn\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:12.924948 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:12.924929    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-credential-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-5n8rn\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:12.925065 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:12.925052    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54l5l\" (UniqueName: \"kubernetes.io/projected/740ca9b8-631f-4381-b864-22b201e29d11-kube-api-access-54l5l\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-5n8rn\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:12.925103 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:12.925095    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-workload-certs\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-5n8rn\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:12.925163 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:12.925148    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/740ca9b8-631f-4381-b864-22b201e29d11-istio-podinfo\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-5n8rn\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:12.925249 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:12.925199    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/740ca9b8-631f-4381-b864-22b201e29d11-istiod-ca-cert\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-5n8rn\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:12.925373 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:12.925359    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-istio-envoy\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-5n8rn\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:13.026581 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.026550    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54l5l\" (UniqueName: \"kubernetes.io/projected/740ca9b8-631f-4381-b864-22b201e29d11-kube-api-access-54l5l\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-5n8rn\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:13.026766 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.026607    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-workload-certs\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-5n8rn\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:13.026766 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.026633    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/740ca9b8-631f-4381-b864-22b201e29d11-istio-podinfo\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-5n8rn\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:13.026766 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.026666    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/740ca9b8-631f-4381-b864-22b201e29d11-istiod-ca-cert\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-5n8rn\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:13.026911 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.026768    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-istio-envoy\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-5n8rn\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:13.026911 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.026873    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/740ca9b8-631f-4381-b864-22b201e29d11-istio-token\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-5n8rn\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:13.027018 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.026955    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-istio-data\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-5n8rn\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:13.027018 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.026995    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-workload-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-5n8rn\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:13.027129 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.027052    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-credential-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-5n8rn\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:13.027129 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.027091    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-workload-certs\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-5n8rn\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:13.027405 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.027376    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-istio-data\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-5n8rn\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:13.027405 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.027395    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-workload-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-5n8rn\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:13.027765 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.027572    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/740ca9b8-631f-4381-b864-22b201e29d11-istiod-ca-cert\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-5n8rn\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:13.027765 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.027572    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-credential-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-5n8rn\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:13.029182 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.029160    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-istio-envoy\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-5n8rn\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:13.029529 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.029506    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/740ca9b8-631f-4381-b864-22b201e29d11-istio-podinfo\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-5n8rn\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:13.039719 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.039696    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/740ca9b8-631f-4381-b864-22b201e29d11-istio-token\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-5n8rn\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:13.040273 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.040256    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54l5l\" (UniqueName: \"kubernetes.io/projected/740ca9b8-631f-4381-b864-22b201e29d11-kube-api-access-54l5l\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-5n8rn\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:13.090849 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.090814    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:13.441195 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.441153    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:10:13.441352 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.441255    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:10:13.441352 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.441289    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:10:13.444690 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.444659    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"]
Nov 03 20:10:13.525219 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.525157    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"]
Nov 03 20:10:13.538739 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.538713    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.540839 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.540814    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"]
Nov 03 20:10:13.542515 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.542482    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.542628 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.542556    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcvx6\" (UniqueName: \"kubernetes.io/projected/a884ccd6-a398-44d2-a3a7-40c8e96abd66-kube-api-access-pcvx6\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.542628 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.542620    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.543117 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.543091    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.543285 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.543141    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.543285 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.543194    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.543503 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.543463    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.543593 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.543519    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.543677 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.543603    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.549257 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.548177    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-46a6d7-openshift-default-dockercfg-5jblx\""
Nov 03 20:10:13.645515 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.645472    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.646341 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.646308    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.646493 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.646348    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.646493 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.646381    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.646493 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.646437    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.646493 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.646458    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.646743 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.646515    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.646743 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.646662    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.646743 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.646695    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcvx6\" (UniqueName: \"kubernetes.io/projected/a884ccd6-a398-44d2-a3a7-40c8e96abd66-kube-api-access-pcvx6\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.646922 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.646809    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.647128 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.647097    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.647594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.647567    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.647700 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.647615    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.647844 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.647822    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.650365 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.650345    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.651367 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.651339    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.659935 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.659914    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcvx6\" (UniqueName: \"kubernetes.io/projected/a884ccd6-a398-44d2-a3a7-40c8e96abd66-kube-api-access-pcvx6\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.662545 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.662520    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:13.909105 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:13.909075    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:14.230568 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:14.230536    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"]
Nov 03 20:10:14.237360 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:10:14.237330    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda884ccd6_a398_44d2_a3a7_40c8e96abd66.slice/crio-a964cf41c0481b9a7dfe6f8d05609563c1cdcbc0658c4993382dfb87295accc5 WatchSource:0}: Error finding container a964cf41c0481b9a7dfe6f8d05609563c1cdcbc0658c4993382dfb87295accc5: Status 404 returned error can't find the container with id a964cf41c0481b9a7dfe6f8d05609563c1cdcbc0658c4993382dfb87295accc5
Nov 03 20:10:14.239917 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:14.239883    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:10:14.240048 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:14.239962    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:10:14.240048 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:14.239997    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:10:14.357165 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:14.357134    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn" event={"ID":"740ca9b8-631f-4381-b864-22b201e29d11","Type":"ContainerStarted","Data":"0800e396985cee475c4e8982d74125b954f8bb4511c4b0f5ff071df2e9e092de"}
Nov 03 20:10:14.357408 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:14.357192    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn" event={"ID":"740ca9b8-631f-4381-b864-22b201e29d11","Type":"ContainerStarted","Data":"a93dac37e039405f21a9097ee47ffd3447350d6449f3777d22b496cdb5c9ac39"}
Nov 03 20:10:14.369362 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:14.362750    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67" event={"ID":"a884ccd6-a398-44d2-a3a7-40c8e96abd66","Type":"ContainerStarted","Data":"a964cf41c0481b9a7dfe6f8d05609563c1cdcbc0658c4993382dfb87295accc5"}
Nov 03 20:10:14.413378 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:14.413304    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn" podStartSLOduration=2.413279557 podStartE2EDuration="2.413279557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:14.409567365 +0000 UTC m=+1963.181860413" watchObservedRunningTime="2026-11-03 20:10:14.413279557 +0000 UTC m=+1963.185572575"
Nov 03 20:10:15.097229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:15.094468    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:15.104229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:15.102345    5591 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Nov 03 20:10:15.104229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:15.102412    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn" podUID="740ca9b8-631f-4381-b864-22b201e29d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Nov 03 20:10:15.372549 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:15.372448    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67" event={"ID":"a884ccd6-a398-44d2-a3a7-40c8e96abd66","Type":"ContainerStarted","Data":"c7b491a1f94b7540bb8bd0aba6f4e2823ae4521ea8a70fd0f3d86374d9c6330b"}
Nov 03 20:10:15.406910 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:15.406858    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67" podStartSLOduration=2.406843196 podStartE2EDuration="2.406843196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:15.405299343 +0000 UTC m=+1964.177592362" watchObservedRunningTime="2026-11-03 20:10:15.406843196 +0000 UTC m=+1964.179136238"
Nov 03 20:10:15.922907 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:15.922872    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:15.924053 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:15.924025    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Nov 03 20:10:15.924309 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:15.924283    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67" podUID="a884ccd6-a398-44d2-a3a7-40c8e96abd66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Nov 03 20:10:16.098230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:16.097629    5591 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Nov 03 20:10:16.098230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:16.097696    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn" podUID="740ca9b8-631f-4381-b864-22b201e29d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Nov 03 20:10:16.265331 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:16.265300    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp"]
Nov 03 20:10:16.270797 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:16.270774    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp"
Nov 03 20:10:16.300234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:16.300005    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-b503a8-dockercfg-d2m4z\""
Nov 03 20:10:16.300234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:16.300146    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-b503a8\""
Nov 03 20:10:16.351279 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:16.350732    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp"]
Nov 03 20:10:16.534292 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:16.533988    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfnv7\" (UniqueName: \"kubernetes.io/projected/2de58be6-f6e0-4367-ae8a-55cdd99ab480-kube-api-access-nfnv7\") pod \"payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp\" (UID: \"2de58be6-f6e0-4367-ae8a-55cdd99ab480\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp"
Nov 03 20:10:16.534292 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:16.534238    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de58be6-f6e0-4367-ae8a-55cdd99ab480-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp\" (UID: \"2de58be6-f6e0-4367-ae8a-55cdd99ab480\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp"
Nov 03 20:10:16.642869 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:16.642830    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfnv7\" (UniqueName: \"kubernetes.io/projected/2de58be6-f6e0-4367-ae8a-55cdd99ab480-kube-api-access-nfnv7\") pod \"payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp\" (UID: \"2de58be6-f6e0-4367-ae8a-55cdd99ab480\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp"
Nov 03 20:10:16.691236 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:16.660584    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de58be6-f6e0-4367-ae8a-55cdd99ab480-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp\" (UID: \"2de58be6-f6e0-4367-ae8a-55cdd99ab480\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp"
Nov 03 20:10:16.716308 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:16.692410    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de58be6-f6e0-4367-ae8a-55cdd99ab480-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp\" (UID: \"2de58be6-f6e0-4367-ae8a-55cdd99ab480\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp"
Nov 03 20:10:16.747234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:16.720178    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfnv7\" (UniqueName: \"kubernetes.io/projected/2de58be6-f6e0-4367-ae8a-55cdd99ab480-kube-api-access-nfnv7\") pod \"payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp\" (UID: \"2de58be6-f6e0-4367-ae8a-55cdd99ab480\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp"
Nov 03 20:10:16.912716 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:16.912618    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Nov 03 20:10:16.920341 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:16.920301    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67" podUID="a884ccd6-a398-44d2-a3a7-40c8e96abd66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Nov 03 20:10:16.966266 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:16.966198    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp"
Nov 03 20:10:17.106232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:17.105009    5591 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Nov 03 20:10:17.106232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:17.105074    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn" podUID="740ca9b8-631f-4381-b864-22b201e29d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Nov 03 20:10:17.371193 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:17.371159    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp"]
Nov 03 20:10:17.373060 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:10:17.373015    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de58be6_f6e0_4367_ae8a_55cdd99ab480.slice/crio-e51bb117ef1ee264f6aa7798f6f11cd1920a3cf84ae5dc6f4027e4b7c8a6378e WatchSource:0}: Error finding container e51bb117ef1ee264f6aa7798f6f11cd1920a3cf84ae5dc6f4027e4b7c8a6378e: Status 404 returned error can't find the container with id e51bb117ef1ee264f6aa7798f6f11cd1920a3cf84ae5dc6f4027e4b7c8a6378e
Nov 03 20:10:17.837351 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:17.837306    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp" event={"ID":"2de58be6-f6e0-4367-ae8a-55cdd99ab480","Type":"ContainerStarted","Data":"5ef82ac9417698e159e614ed07d3fd1c4a554fe6e1cec66b4375c06673a4b30e"}
Nov 03 20:10:17.837594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:17.837372    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp" event={"ID":"2de58be6-f6e0-4367-ae8a-55cdd99ab480","Type":"ContainerStarted","Data":"e51bb117ef1ee264f6aa7798f6f11cd1920a3cf84ae5dc6f4027e4b7c8a6378e"}
Nov 03 20:10:17.837594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:17.837538    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp"
Nov 03 20:10:17.859923 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:17.859872    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp" podStartSLOduration=1.85985736 podStartE2EDuration="1.85985736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:17.855327298 +0000 UTC m=+1966.627620317" watchObservedRunningTime="2026-11-03 20:10:17.85985736 +0000 UTC m=+1966.632150382"
Nov 03 20:10:17.911290 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:17.910532    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Nov 03 20:10:17.911290 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:17.910596    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67" podUID="a884ccd6-a398-44d2-a3a7-40c8e96abd66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Nov 03 20:10:18.099232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:18.096628    5591 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Nov 03 20:10:18.099232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:18.096694    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn" podUID="740ca9b8-631f-4381-b864-22b201e29d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Nov 03 20:10:18.910231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:18.909730    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Nov 03 20:10:18.910231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:18.909793    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67" podUID="a884ccd6-a398-44d2-a3a7-40c8e96abd66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Nov 03 20:10:19.100231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:19.094705    5591 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Nov 03 20:10:19.100231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:19.094795    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn" podUID="740ca9b8-631f-4381-b864-22b201e29d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Nov 03 20:10:19.915232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:19.914432    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Nov 03 20:10:19.915232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:19.914492    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67" podUID="a884ccd6-a398-44d2-a3a7-40c8e96abd66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Nov 03 20:10:20.098498 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:20.098461    5591 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body=
Nov 03 20:10:20.098772 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:20.098746    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn" podUID="740ca9b8-631f-4381-b864-22b201e29d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused"
Nov 03 20:10:20.909522 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:20.909484    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body=
Nov 03 20:10:20.909750 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:20.909552    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67" podUID="a884ccd6-a398-44d2-a3a7-40c8e96abd66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused"
Nov 03 20:10:21.109417 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:21.109386    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:21.110726 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:21.110606    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:21.111567 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:21.111545    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:10:21.935104 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:21.932259    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:21.935104 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:21.934185    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:21.937120 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:21.937096    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:10:27.369189 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:27.369149    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:10:28.891425 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:28.891392    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp"
Nov 03 20:10:28.935630 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:10:28.935591    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:10:35.545932 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:35.545869    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf"]
Nov 03 20:10:35.553587 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:35.553559    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf"
Nov 03 20:10:35.566507 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:35.566466    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-b503a8-serving-cert\""
Nov 03 20:10:35.572572 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:35.572545    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf"]
Nov 03 20:10:35.651715 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:35.651679    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c175022-5f51-4cce-8f64-bb399a11ce8d-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf\" (UID: \"8c175022-5f51-4cce-8f64-bb399a11ce8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf"
Nov 03 20:10:35.651958 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:35.651863    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-655m2\" (UniqueName: \"kubernetes.io/projected/8c175022-5f51-4cce-8f64-bb399a11ce8d-kube-api-access-655m2\") pod \"maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf\" (UID: \"8c175022-5f51-4cce-8f64-bb399a11ce8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf"
Nov 03 20:10:35.652040 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:35.651959    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c175022-5f51-4cce-8f64-bb399a11ce8d-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf\" (UID: \"8c175022-5f51-4cce-8f64-bb399a11ce8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf"
Nov 03 20:10:35.652040 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:35.651992    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c175022-5f51-4cce-8f64-bb399a11ce8d-metrics-certs\") pod \"maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf\" (UID: \"8c175022-5f51-4cce-8f64-bb399a11ce8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf"
Nov 03 20:10:35.761227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:35.758958    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c175022-5f51-4cce-8f64-bb399a11ce8d-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf\" (UID: \"8c175022-5f51-4cce-8f64-bb399a11ce8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf"
Nov 03 20:10:35.761227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:35.759114    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-655m2\" (UniqueName: \"kubernetes.io/projected/8c175022-5f51-4cce-8f64-bb399a11ce8d-kube-api-access-655m2\") pod \"maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf\" (UID: \"8c175022-5f51-4cce-8f64-bb399a11ce8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf"
Nov 03 20:10:35.761227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:35.759367    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c175022-5f51-4cce-8f64-bb399a11ce8d-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf\" (UID: \"8c175022-5f51-4cce-8f64-bb399a11ce8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf"
Nov 03 20:10:35.761227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:35.759423    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c175022-5f51-4cce-8f64-bb399a11ce8d-metrics-certs\") pod \"maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf\" (UID: \"8c175022-5f51-4cce-8f64-bb399a11ce8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf"
Nov 03 20:10:35.791183 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:35.787333    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c175022-5f51-4cce-8f64-bb399a11ce8d-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf\" (UID: \"8c175022-5f51-4cce-8f64-bb399a11ce8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf"
Nov 03 20:10:35.825282 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:35.819816    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-655m2\" (UniqueName: \"kubernetes.io/projected/8c175022-5f51-4cce-8f64-bb399a11ce8d-kube-api-access-655m2\") pod \"maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf\" (UID: \"8c175022-5f51-4cce-8f64-bb399a11ce8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf"
Nov 03 20:10:35.836256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:35.834936    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c175022-5f51-4cce-8f64-bb399a11ce8d-maas-api-tls\") pod \"maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf\" (UID: \"8c175022-5f51-4cce-8f64-bb399a11ce8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf"
Nov 03 20:10:35.836256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:35.835663    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c175022-5f51-4cce-8f64-bb399a11ce8d-metrics-certs\") pod \"maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf\" (UID: \"8c175022-5f51-4cce-8f64-bb399a11ce8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf"
Nov 03 20:10:35.885152 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:35.885114    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf"
Nov 03 20:10:36.174067 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:36.174031    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf"]
Nov 03 20:10:36.174197 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:10:36.174128    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c175022_5f51_4cce_8f64_bb399a11ce8d.slice/crio-d9c20fa8de58d2f56f06131727c59e2a2575ec01b7d52f3ed0d11715c5b757e7 WatchSource:0}: Error finding container d9c20fa8de58d2f56f06131727c59e2a2575ec01b7d52f3ed0d11715c5b757e7: Status 404 returned error can't find the container with id d9c20fa8de58d2f56f06131727c59e2a2575ec01b7d52f3ed0d11715c5b757e7
Nov 03 20:10:37.081776 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:37.081740    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf" event={"ID":"8c175022-5f51-4cce-8f64-bb399a11ce8d","Type":"ContainerStarted","Data":"206495bb49b72f152ad55b326fa84144d13813ece464c80556b5f7262ead93a4"}
Nov 03 20:10:37.082345 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:37.081805    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf" event={"ID":"8c175022-5f51-4cce-8f64-bb399a11ce8d","Type":"ContainerStarted","Data":"d9c20fa8de58d2f56f06131727c59e2a2575ec01b7d52f3ed0d11715c5b757e7"}
Nov 03 20:10:37.082345 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:37.081840    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf"
Nov 03 20:10:37.102022 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:37.101956    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf" podStartSLOduration=1.63550481 podStartE2EDuration="2.10193898s" totalImagesPullingTime="466.43417ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:37.101622162 +0000 UTC m=+1985.873915182" watchObservedRunningTime="2026-11-03 20:10:37.10193898 +0000 UTC m=+1985.874231999"
Nov 03 20:10:43.095824 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:43.095753    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf"
Nov 03 20:10:57.756548 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:57.756514    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-36556d-f66cd46b-fz9td"]
Nov 03 20:10:57.784315 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:57.784279    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k"]
Nov 03 20:10:57.840991 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:57.840955    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"]
Nov 03 20:10:57.880707 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:57.880667    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"]
Nov 03 20:10:57.880923 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:57.880821    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:57.984916 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:57.984880    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-envoy\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nmhgk\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:57.985092 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:57.984995    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-token\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nmhgk\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:57.985092 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:57.985071    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-credential-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nmhgk\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:57.985243 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:57.985226    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-data\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nmhgk\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:57.985295 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:57.985277    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-workload-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nmhgk\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:57.985348 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:57.985337    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istiod-ca-cert\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nmhgk\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:57.985417 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:57.985398    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-podinfo\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nmhgk\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:57.985526 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:57.985513    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-workload-certs\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nmhgk\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:57.985572 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:57.985547    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4824q\" (UniqueName: \"kubernetes.io/projected/e2d939a4-ec3b-4a1a-8060-cc73381fc711-kube-api-access-4824q\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nmhgk\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:58.095265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.091398    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-workload-certs\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nmhgk\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:58.095265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.091479    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4824q\" (UniqueName: \"kubernetes.io/projected/e2d939a4-ec3b-4a1a-8060-cc73381fc711-kube-api-access-4824q\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nmhgk\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:58.095265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.091536    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-envoy\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nmhgk\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:58.095265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.091604    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-token\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nmhgk\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:58.095265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.091653    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-credential-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nmhgk\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:58.095265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.091823    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-data\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nmhgk\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:58.095265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.091855    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-workload-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nmhgk\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:58.095265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.091941    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istiod-ca-cert\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nmhgk\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:58.095265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.091991    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-podinfo\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nmhgk\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:58.115294 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.100589    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-workload-certs\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nmhgk\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:58.115557 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.103827    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-podinfo\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nmhgk\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:58.115679 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.104119    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-credential-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nmhgk\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:58.115772 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.113045    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istiod-ca-cert\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nmhgk\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:58.115997 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.113609    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-data\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nmhgk\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:58.148235 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.134993    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-workload-socket\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nmhgk\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:58.153546 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.153503    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-envoy\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nmhgk\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:58.154295 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.154264    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-token\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nmhgk\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:58.173230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.162995    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4824q\" (UniqueName: \"kubernetes.io/projected/e2d939a4-ec3b-4a1a-8060-cc73381fc711-kube-api-access-4824q\") pod \"e2e-autores-36556d-openshift-default-658cbc6465-nmhgk\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") " pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:58.208705 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.208669    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:10:58.693635 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.687157    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"]
Nov 03 20:10:58.709110 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.709085    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:58.728051 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.726523    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"]
Nov 03 20:10:58.764114 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.757305    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"]
Nov 03 20:10:58.764114 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.763638    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:58.795259 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.790655    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"]
Nov 03 20:10:58.839275 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:10:58.836550    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:10:58.885256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.884374    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:58.885256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.884442    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:58.885256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.884507    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:58.885256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.884562    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:58.885256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.884670    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d98020b-67af-47b0-85fc-0190861a7d37-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:58.885256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.884705    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:58.885256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.884763    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:58.885256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.884791    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:58.885256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.884823    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:58.885256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.884864    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d98020b-67af-47b0-85fc-0190861a7d37-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:58.885256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.884939    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:58.885256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.884968    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:58.885256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.885058    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:58.885256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.885168    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zgg8\" (UniqueName: \"kubernetes.io/projected/9d98020b-67af-47b0-85fc-0190861a7d37-kube-api-access-9zgg8\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:58.885256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.885219    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d98020b-67af-47b0-85fc-0190861a7d37-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:58.886167 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.885277    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkz9q\" (UniqueName: \"kubernetes.io/projected/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-kube-api-access-gkz9q\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:58.886167 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.885325    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:58.886167 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:58.885361    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:58.990492 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:10:58.954316    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d939a4_ec3b_4a1a_8060_cc73381fc711.slice/crio-9324704bc7b22c4dd1bed02e78097a4c5ef5606628176c9c0a1c6b0bdcf60e25 WatchSource:0}: Error finding container 9324704bc7b22c4dd1bed02e78097a4c5ef5606628176c9c0a1c6b0bdcf60e25: Status 404 returned error can't find the container with id 9324704bc7b22c4dd1bed02e78097a4c5ef5606628176c9c0a1c6b0bdcf60e25
Nov 03 20:10:59.065229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.055327    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkz9q\" (UniqueName: \"kubernetes.io/projected/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-kube-api-access-gkz9q\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:59.065229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.055391    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:59.065229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.055424    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:59.065229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.055472    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:59.065229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.055507    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:59.065229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.055562    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:59.065229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.055614    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:59.065229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.055724    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d98020b-67af-47b0-85fc-0190861a7d37-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:59.065229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.055760    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:59.065229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.055803    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:59.065229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.055833    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:59.065229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.055865    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:59.065229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.055907    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d98020b-67af-47b0-85fc-0190861a7d37-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:59.065229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.055981    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:59.065229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.056008    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:59.065229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.056098    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:59.065229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.056227    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zgg8\" (UniqueName: \"kubernetes.io/projected/9d98020b-67af-47b0-85fc-0190861a7d37-kube-api-access-9zgg8\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:59.065229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.056258    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d98020b-67af-47b0-85fc-0190861a7d37-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:59.065229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.057044    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d98020b-67af-47b0-85fc-0190861a7d37-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:59.163487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.073047    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:10:59.163487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.073151    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:10:59.163487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.100424    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:59.163487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.108340    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"]
Nov 03 20:10:59.163487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.114879    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:59.163487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.115557    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istiod-ca-cert\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:59.163487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.115803    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:59.163487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.116015    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-workload-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:59.163487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.118543    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:59.163487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.119344    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-data\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:59.163487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.119612    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-credential-socket\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:59.163487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.119847    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-workload-certs\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:59.163487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.120361    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:10:59.163487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.155768    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:59.302037 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.262572    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d98020b-67af-47b0-85fc-0190861a7d37-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:59.302037 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.276359    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d98020b-67af-47b0-85fc-0190861a7d37-istio-token\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:59.302037 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.277277    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-podinfo\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:59.302037 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.278937    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:59.302037 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.279285    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-istio-envoy\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:59.302037 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.279635    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkz9q\" (UniqueName: \"kubernetes.io/projected/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-kube-api-access-gkz9q\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:59.302037 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.280297    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zgg8\" (UniqueName: \"kubernetes.io/projected/9d98020b-67af-47b0-85fc-0190861a7d37-kube-api-access-9zgg8\") pod \"e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") " pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:59.460992 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.332284    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:10:59.497799 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.475923    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:10:59.588438 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.588355    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk" event={"ID":"e2d939a4-ec3b-4a1a-8060-cc73381fc711","Type":"ContainerStarted","Data":"9324704bc7b22c4dd1bed02e78097a4c5ef5606628176c9c0a1c6b0bdcf60e25"}
Nov 03 20:10:59.787317 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.774269    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz"]
Nov 03 20:10:59.968517 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.962451    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz"]
Nov 03 20:10:59.968517 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.962589    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz"
Nov 03 20:10:59.985013 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.984984    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"routing-overlay\""
Nov 03 20:10:59.985123 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.985039    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"payload-processing-external-model-e2e--b1878920-dockercfg-6sqzm\""
Nov 03 20:10:59.985221 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.985132    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"agc-cred-6e3d6ce11b17\""
Nov 03 20:10:59.985221 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.985190    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0e4ce7\""
Nov 03 20:10:59.985814 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.985761    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"kube-root-ca.crt\""
Nov 03 20:10:59.987410 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:10:59.987160    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"openshift-service-ca.crt\""
Nov 03 20:11:00.055227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.055107    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/774a0a2c-2534-4a5c-9c2d-737b3aef7268-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz\" (UID: \"774a0a2c-2534-4a5c-9c2d-737b3aef7268\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz"
Nov 03 20:11:00.055461 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.055422    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/774a0a2c-2534-4a5c-9c2d-737b3aef7268-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz\" (UID: \"774a0a2c-2534-4a5c-9c2d-737b3aef7268\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz"
Nov 03 20:11:00.055551 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.055479    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdk55\" (UniqueName: \"kubernetes.io/projected/774a0a2c-2534-4a5c-9c2d-737b3aef7268-kube-api-access-rdk55\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz\" (UID: \"774a0a2c-2534-4a5c-9c2d-737b3aef7268\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz"
Nov 03 20:11:00.055625 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.055594    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/774a0a2c-2534-4a5c-9c2d-737b3aef7268-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz\" (UID: \"774a0a2c-2534-4a5c-9c2d-737b3aef7268\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz"
Nov 03 20:11:00.111787 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:11:00.111742    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d98020b_67af_47b0_85fc_0190861a7d37.slice/crio-91901be83901b1893145ebfd625a4dc7cc82a0241ae031987e4f40e930e52ff9 WatchSource:0}: Error finding container 91901be83901b1893145ebfd625a4dc7cc82a0241ae031987e4f40e930e52ff9: Status 404 returned error can't find the container with id 91901be83901b1893145ebfd625a4dc7cc82a0241ae031987e4f40e930e52ff9
Nov 03 20:11:00.113284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.113253    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"]
Nov 03 20:11:00.115957 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.115915    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:11:00.116068 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.116007    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:11:00.116068 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.116046    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:11:00.157306 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.157274    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/774a0a2c-2534-4a5c-9c2d-737b3aef7268-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz\" (UID: \"774a0a2c-2534-4a5c-9c2d-737b3aef7268\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz"
Nov 03 20:11:00.158475 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.158450    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/774a0a2c-2534-4a5c-9c2d-737b3aef7268-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz\" (UID: \"774a0a2c-2534-4a5c-9c2d-737b3aef7268\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz"
Nov 03 20:11:00.158599 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.158527    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/774a0a2c-2534-4a5c-9c2d-737b3aef7268-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz\" (UID: \"774a0a2c-2534-4a5c-9c2d-737b3aef7268\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz"
Nov 03 20:11:00.158685 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.158624    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdk55\" (UniqueName: \"kubernetes.io/projected/774a0a2c-2534-4a5c-9c2d-737b3aef7268-kube-api-access-rdk55\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz\" (UID: \"774a0a2c-2534-4a5c-9c2d-737b3aef7268\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz"
Nov 03 20:11:00.158793 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.158760    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/774a0a2c-2534-4a5c-9c2d-737b3aef7268-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz\" (UID: \"774a0a2c-2534-4a5c-9c2d-737b3aef7268\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz"
Nov 03 20:11:00.159389 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.159367    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/774a0a2c-2534-4a5c-9c2d-737b3aef7268-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz\" (UID: \"774a0a2c-2534-4a5c-9c2d-737b3aef7268\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz"
Nov 03 20:11:00.160439 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.160420    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/774a0a2c-2534-4a5c-9c2d-737b3aef7268-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz\" (UID: \"774a0a2c-2534-4a5c-9c2d-737b3aef7268\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz"
Nov 03 20:11:00.174662 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.174640    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdk55\" (UniqueName: \"kubernetes.io/projected/774a0a2c-2534-4a5c-9c2d-737b3aef7268-kube-api-access-rdk55\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz\" (UID: \"774a0a2c-2534-4a5c-9c2d-737b3aef7268\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz"
Nov 03 20:11:00.206470 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.206426    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"]
Nov 03 20:11:00.208045 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:11:00.208018    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a3c3c2e_65cc_4468_b7bb_3f65107b2e9b.slice/crio-ec9ac6c08980f43f5b1bf95ad776c0c308cab9b489a88fd7eb7a45f8cc29e04a WatchSource:0}: Error finding container ec9ac6c08980f43f5b1bf95ad776c0c308cab9b489a88fd7eb7a45f8cc29e04a: Status 404 returned error can't find the container with id ec9ac6c08980f43f5b1bf95ad776c0c308cab9b489a88fd7eb7a45f8cc29e04a
Nov 03 20:11:00.211377 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.211343    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:11:00.211486 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.211428    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:11:00.211486 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.211478    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:11:00.307794 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.307707    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz"
Nov 03 20:11:00.646544 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.646453    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p" event={"ID":"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b","Type":"ContainerStarted","Data":"007c05e37eb70000c378f832736aceb97542b9853f342da2754550f23438f9e1"}
Nov 03 20:11:00.646544 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.646513    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p" event={"ID":"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b","Type":"ContainerStarted","Data":"ec9ac6c08980f43f5b1bf95ad776c0c308cab9b489a88fd7eb7a45f8cc29e04a"}
Nov 03 20:11:00.649511 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.649422    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6" event={"ID":"9d98020b-67af-47b0-85fc-0190861a7d37","Type":"ContainerStarted","Data":"3fd856d3df1ad93e35540a9c66f8d6b24e9df7a6bfef4ddabfda520f07793bf2"}
Nov 03 20:11:00.649511 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.649478    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6" event={"ID":"9d98020b-67af-47b0-85fc-0190861a7d37","Type":"ContainerStarted","Data":"91901be83901b1893145ebfd625a4dc7cc82a0241ae031987e4f40e930e52ff9"}
Nov 03 20:11:00.651861 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.651839    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk" event={"ID":"e2d939a4-ec3b-4a1a-8060-cc73381fc711","Type":"ContainerStarted","Data":"f71692c47e3066cfe6f1ca749e1bce6fef38a4d872ac9f0a1d849c954a9569fb"}
Nov 03 20:11:00.668872 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.668664    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz"]
Nov 03 20:11:00.669395 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:11:00.669367    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774a0a2c_2534_4a5c_9c2d_737b3aef7268.slice/crio-6bbdc22aef988ece39856be4219066b553a875fc9924cbdf7a76b28e2b65a1df WatchSource:0}: Error finding container 6bbdc22aef988ece39856be4219066b553a875fc9924cbdf7a76b28e2b65a1df: Status 404 returned error can't find the container with id 6bbdc22aef988ece39856be4219066b553a875fc9924cbdf7a76b28e2b65a1df
Nov 03 20:11:00.737574 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.737510    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p" podStartSLOduration=2.7374875100000002 podStartE2EDuration="2.73748751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:00.676518171 +0000 UTC m=+2009.448811201" watchObservedRunningTime="2026-11-03 20:11:00.73748751 +0000 UTC m=+2009.509780529"
Nov 03 20:11:00.806114 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.806028    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6" podStartSLOduration=2.80598367 podStartE2EDuration="2.80598367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:00.702740064 +0000 UTC m=+2009.475033093" watchObservedRunningTime="2026-11-03 20:11:00.80598367 +0000 UTC m=+2009.578276688"
Nov 03 20:11:00.877321 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:00.877254    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk" podStartSLOduration=3.877231519 podStartE2EDuration="3.877231519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:00.761692006 +0000 UTC m=+2009.533985039" watchObservedRunningTime="2026-11-03 20:11:00.877231519 +0000 UTC m=+2009.649524539"
Nov 03 20:11:01.255354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:01.255258    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:11:01.257371 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:01.257338    5591 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Nov 03 20:11:01.257639 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:01.257590    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk" podUID="e2d939a4-ec3b-4a1a-8060-cc73381fc711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Nov 03 20:11:01.359983 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:01.359922    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 03 20:11:01.360314 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:01.360282    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6" podUID="9d98020b-67af-47b0-85fc-0190861a7d37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 03 20:11:01.375589 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:01.375563    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:11:01.505237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:01.505181    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:11:01.506956 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:01.506877    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Nov 03 20:11:01.507154 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:01.507129    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p" podUID="1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Nov 03 20:11:02.026231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:02.018628    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz" event={"ID":"774a0a2c-2534-4a5c-9c2d-737b3aef7268","Type":"ContainerStarted","Data":"092b1191e6d4c34ad8fc0c0e892c2b6a6358097fdf82913d1ab40599b5094a6c"}
Nov 03 20:11:02.034341 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:02.034306    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz" event={"ID":"774a0a2c-2534-4a5c-9c2d-737b3aef7268","Type":"ContainerStarted","Data":"6bbdc22aef988ece39856be4219066b553a875fc9924cbdf7a76b28e2b65a1df"}
Nov 03 20:11:02.089346 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:02.089316    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz"
Nov 03 20:11:02.225986 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:02.225950    5591 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Nov 03 20:11:02.235518 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:02.235476    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk" podUID="e2d939a4-ec3b-4a1a-8060-cc73381fc711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Nov 03 20:11:02.338901 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:02.338806    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 03 20:11:02.339185 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:02.339154    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6" podUID="9d98020b-67af-47b0-85fc-0190861a7d37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 03 20:11:02.506045 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:02.506004    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Nov 03 20:11:02.506353 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:02.506319    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p" podUID="1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Nov 03 20:11:02.757555 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:02.757457    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-36556d-f66cd46b-fz9td" podUID="c76d28c6-0efd-4c21-bcf8-ed01189327b0" containerName="payload-processing" containerID="cri-o://1a0d5b107d312836e45c7e57dbb5f7a3047c1c1fbbf7c72994cbd0831792f3f7" gracePeriod=25
Nov 03 20:11:02.793233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:02.787258    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k" podUID="ad53b126-3346-493f-bf9e-139e08b50cf5" containerName="payload-pre-processing" containerID="cri-o://197468be71b58f838fae5ea39a3cffe8d90a90619fc5175bcdbc1aec4180d143" gracePeriod=25
Nov 03 20:11:03.066574 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:03.066536    5591 generic.go:356] "Generic (PLEG): container finished" podID="ad53b126-3346-493f-bf9e-139e08b50cf5" containerID="197468be71b58f838fae5ea39a3cffe8d90a90619fc5175bcdbc1aec4180d143" exitCode=0
Nov 03 20:11:03.067138 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:03.066765    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k" event={"ID":"ad53b126-3346-493f-bf9e-139e08b50cf5","Type":"ContainerDied","Data":"197468be71b58f838fae5ea39a3cffe8d90a90619fc5175bcdbc1aec4180d143"}
Nov 03 20:11:03.069444 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:03.069413    5591 generic.go:356] "Generic (PLEG): container finished" podID="c76d28c6-0efd-4c21-bcf8-ed01189327b0" containerID="1a0d5b107d312836e45c7e57dbb5f7a3047c1c1fbbf7c72994cbd0831792f3f7" exitCode=0
Nov 03 20:11:03.071085 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:03.070400    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-36556d-f66cd46b-fz9td" event={"ID":"c76d28c6-0efd-4c21-bcf8-ed01189327b0","Type":"ContainerDied","Data":"1a0d5b107d312836e45c7e57dbb5f7a3047c1c1fbbf7c72994cbd0831792f3f7"}
Nov 03 20:11:03.225089 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:03.224606    5591 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Nov 03 20:11:03.225089 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:03.224676    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk" podUID="e2d939a4-ec3b-4a1a-8060-cc73381fc711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Nov 03 20:11:03.596372 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:03.589947    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Nov 03 20:11:03.596372 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:03.590000    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p" podUID="1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Nov 03 20:11:03.663449 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:03.660424    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 03 20:11:03.663449 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:03.660485    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6" podUID="9d98020b-67af-47b0-85fc-0190861a7d37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 03 20:11:04.174803 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:04.174782    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-36556d-f66cd46b-fz9td"
Nov 03 20:11:04.254121 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:04.254024    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-36556d-f66cd46b-fz9td" event={"ID":"c76d28c6-0efd-4c21-bcf8-ed01189327b0","Type":"ContainerDied","Data":"d52b5d7fdd2988f2b72504011792b41351252ee0545890b6ef333ef48d11a85e"}
Nov 03 20:11:04.254461 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:04.254438    5591 scope.go:122] "RemoveContainer" containerID="1a0d5b107d312836e45c7e57dbb5f7a3047c1c1fbbf7c72994cbd0831792f3f7"
Nov 03 20:11:04.260454 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:04.257601    5591 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Nov 03 20:11:04.260454 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:04.257656    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk" podUID="e2d939a4-ec3b-4a1a-8060-cc73381fc711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Nov 03 20:11:04.263604 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:04.263572    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k" event={"ID":"ad53b126-3346-493f-bf9e-139e08b50cf5","Type":"ContainerDied","Data":"564105209f811ee847cf6ef715c6763d4aeecbbd015f4f4bd357e0c60333ca7a"}
Nov 03 20:11:04.263741 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:04.263635    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="564105209f811ee847cf6ef715c6763d4aeecbbd015f4f4bd357e0c60333ca7a"
Nov 03 20:11:04.265047 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:04.265027    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k"
Nov 03 20:11:04.268449 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:04.268403    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz" podStartSLOduration=5.268387591 podStartE2EDuration="5.268387591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:02.101638137 +0000 UTC m=+2010.873931146" watchObservedRunningTime="2026-11-03 20:11:04.268387591 +0000 UTC m=+2013.040680612"
Nov 03 20:11:04.355961 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:04.355928    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 03 20:11:04.356260 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:04.356232    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6" podUID="9d98020b-67af-47b0-85fc-0190861a7d37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 03 20:11:04.370930 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:04.370899    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad53b126-3346-493f-bf9e-139e08b50cf5-kube-api-access-z6mlb\" (UniqueName: \"kubernetes.io/projected/ad53b126-3346-493f-bf9e-139e08b50cf5-kube-api-access-z6mlb\") pod \"ad53b126-3346-493f-bf9e-139e08b50cf5\" (UID: \"ad53b126-3346-493f-bf9e-139e08b50cf5\") "
Nov 03 20:11:04.375048 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:04.375022    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c76d28c6-0efd-4c21-bcf8-ed01189327b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76d28c6-0efd-4c21-bcf8-ed01189327b0-plugins-config-volume\") pod \"c76d28c6-0efd-4c21-bcf8-ed01189327b0\" (UID: \"c76d28c6-0efd-4c21-bcf8-ed01189327b0\") "
Nov 03 20:11:04.375273 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:04.375254    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad53b126-3346-493f-bf9e-139e08b50cf5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad53b126-3346-493f-bf9e-139e08b50cf5-plugins-config-volume\") pod \"ad53b126-3346-493f-bf9e-139e08b50cf5\" (UID: \"ad53b126-3346-493f-bf9e-139e08b50cf5\") "
Nov 03 20:11:04.375877 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:04.375857    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c76d28c6-0efd-4c21-bcf8-ed01189327b0-kube-api-access-rrkhv\" (UniqueName: \"kubernetes.io/projected/c76d28c6-0efd-4c21-bcf8-ed01189327b0-kube-api-access-rrkhv\") pod \"c76d28c6-0efd-4c21-bcf8-ed01189327b0\" (UID: \"c76d28c6-0efd-4c21-bcf8-ed01189327b0\") "
Nov 03 20:11:04.376087 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:04.374855    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad53b126-3346-493f-bf9e-139e08b50cf5-kube-api-access-z6mlb" pod "ad53b126-3346-493f-bf9e-139e08b50cf5" (UID: "ad53b126-3346-493f-bf9e-139e08b50cf5"). InnerVolumeSpecName "kube-api-access-z6mlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:04.376173 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:04.375494    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c76d28c6-0efd-4c21-bcf8-ed01189327b0-plugins-config-volume" pod "c76d28c6-0efd-4c21-bcf8-ed01189327b0" (UID: "c76d28c6-0efd-4c21-bcf8-ed01189327b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:11:04.376173 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:04.375780    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad53b126-3346-493f-bf9e-139e08b50cf5-plugins-config-volume" pod "ad53b126-3346-493f-bf9e-139e08b50cf5" (UID: "ad53b126-3346-493f-bf9e-139e08b50cf5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:11:04.386821 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:04.386788    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76d28c6-0efd-4c21-bcf8-ed01189327b0-kube-api-access-rrkhv" pod "c76d28c6-0efd-4c21-bcf8-ed01189327b0" (UID: "c76d28c6-0efd-4c21-bcf8-ed01189327b0"). InnerVolumeSpecName "kube-api-access-rrkhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:04.421487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:04.414608    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6mlb\" (UniqueName: \"kubernetes.io/projected/ad53b126-3346-493f-bf9e-139e08b50cf5-kube-api-access-z6mlb\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:04.421487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:04.414645    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76d28c6-0efd-4c21-bcf8-ed01189327b0-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:04.421487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:04.414660    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad53b126-3346-493f-bf9e-139e08b50cf5-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:04.421487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:04.414676    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrkhv\" (UniqueName: \"kubernetes.io/projected/c76d28c6-0efd-4c21-bcf8-ed01189327b0-kube-api-access-rrkhv\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:04.507786 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:04.507694    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Nov 03 20:11:04.509370 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:04.509336    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p" podUID="1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Nov 03 20:11:05.219750 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:05.219652    5591 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Nov 03 20:11:05.223364 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:05.223326    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk" podUID="e2d939a4-ec3b-4a1a-8060-cc73381fc711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Nov 03 20:11:05.273450 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:05.273421    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-36556d-f66cd46b-fz9td"
Nov 03 20:11:05.273846 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:05.273824    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k"
Nov 03 20:11:05.362580 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:05.348603    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 03 20:11:05.362580 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:05.348658    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6" podUID="9d98020b-67af-47b0-85fc-0190861a7d37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 03 20:11:05.462678 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:05.462647    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k"]
Nov 03 20:11:05.466939 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:05.466919    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-36556d-58f7c56d79-psw9k"]
Nov 03 20:11:05.471546 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:05.471489    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-36556d-f66cd46b-fz9td"]
Nov 03 20:11:05.474972 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:05.474951    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-36556d-f66cd46b-fz9td"]
Nov 03 20:11:05.513230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:05.513112    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Nov 03 20:11:05.513230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:05.513168    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p" podUID="1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Nov 03 20:11:06.212772 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:06.212281    5591 patch_prober.go:29] interesting pod/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Nov 03 20:11:06.212772 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:06.212359    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk" podUID="e2d939a4-ec3b-4a1a-8060-cc73381fc711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Nov 03 20:11:06.337613 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:06.337573    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 03 20:11:06.338369 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:06.338335    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6" podUID="9d98020b-67af-47b0-85fc-0190861a7d37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 03 20:11:06.505377 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:06.505281    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Nov 03 20:11:06.506110 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:06.506075    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p" podUID="1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Nov 03 20:11:07.233533 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:07.233290    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:11:07.236624 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:07.235065    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:11:07.236624 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:07.235640    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:11:07.338772 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:07.338143    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body=
Nov 03 20:11:07.338772 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:07.338249    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6" podUID="9d98020b-67af-47b0-85fc-0190861a7d37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused"
Nov 03 20:11:07.354087 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:07.353859    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad53b126-3346-493f-bf9e-139e08b50cf5" path="/var/lib/kubelet/pods/ad53b126-3346-493f-bf9e-139e08b50cf5/volumes"
Nov 03 20:11:07.354418 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:07.354403    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c76d28c6-0efd-4c21-bcf8-ed01189327b0" path="/var/lib/kubelet/pods/c76d28c6-0efd-4c21-bcf8-ed01189327b0/volumes"
Nov 03 20:11:07.505679 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:07.505161    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Nov 03 20:11:07.505679 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:07.505239    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p" podUID="1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Nov 03 20:11:08.343148 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:08.343111    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:11:08.343643 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:08.343613    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:11:08.344371 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:08.344349    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:11:08.553608 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:08.535018    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:11:08.553608 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:08.536080    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:11:08.560697 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:08.560656    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:11:14.266801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.266767    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz"
Nov 03 20:11:14.343102 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.343069    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf"]
Nov 03 20:11:14.344069 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.344049    5591 cpu_manager.go:414] "containerMap: removing container" podUID="ad53b126-3346-493f-bf9e-139e08b50cf5" containerName="payload-pre-processing"
Nov 03 20:11:14.344069 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.344070    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad53b126-3346-493f-bf9e-139e08b50cf5" containerName="payload-pre-processing"
Nov 03 20:11:14.344174 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.344090    5591 cpu_manager.go:414] "containerMap: removing container" podUID="c76d28c6-0efd-4c21-bcf8-ed01189327b0" containerName="payload-processing"
Nov 03 20:11:14.344174 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.344096    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c76d28c6-0efd-4c21-bcf8-ed01189327b0" containerName="payload-processing"
Nov 03 20:11:14.344302 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.344284    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad53b126-3346-493f-bf9e-139e08b50cf5" containerName="payload-pre-processing"
Nov 03 20:11:14.344302 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.344297    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c76d28c6-0efd-4c21-bcf8-ed01189327b0" containerName="payload-processing"
Nov 03 20:11:14.350263 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.350224    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf"
Nov 03 20:11:14.361910 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.361873    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf"]
Nov 03 20:11:14.362898 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.362868    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b503a8\"/\"agc-cred-5735d3dde484\""
Nov 03 20:11:14.363784 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.363691    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b503a8\"/\"routing-overlay\""
Nov 03 20:11:14.363784 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.363771    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b503a8\"/\"openshift-service-ca.crt\""
Nov 03 20:11:14.363973 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.363861    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b503a8\"/\"payload-processing-external-model-plugins-e2e-extmod-b-b503a8\""
Nov 03 20:11:14.365500 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.365337    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b503a8\"/\"kube-root-ca.crt\""
Nov 03 20:11:14.441307 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.441258    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c30c58b-4263-430d-acd1-0b93599407c6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf\" (UID: \"2c30c58b-4263-430d-acd1-0b93599407c6\") " pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf"
Nov 03 20:11:14.441507 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.441467    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c30c58b-4263-430d-acd1-0b93599407c6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf\" (UID: \"2c30c58b-4263-430d-acd1-0b93599407c6\") " pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf"
Nov 03 20:11:14.441624 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.441603    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c30c58b-4263-430d-acd1-0b93599407c6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf\" (UID: \"2c30c58b-4263-430d-acd1-0b93599407c6\") " pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf"
Nov 03 20:11:14.441726 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.441706    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjlqv\" (UniqueName: \"kubernetes.io/projected/2c30c58b-4263-430d-acd1-0b93599407c6-kube-api-access-fjlqv\") pod \"payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf\" (UID: \"2c30c58b-4263-430d-acd1-0b93599407c6\") " pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf"
Nov 03 20:11:14.563172 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.563067    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c30c58b-4263-430d-acd1-0b93599407c6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf\" (UID: \"2c30c58b-4263-430d-acd1-0b93599407c6\") " pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf"
Nov 03 20:11:14.575357 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.566162    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c30c58b-4263-430d-acd1-0b93599407c6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf\" (UID: \"2c30c58b-4263-430d-acd1-0b93599407c6\") " pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf"
Nov 03 20:11:14.578668 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.578635    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c30c58b-4263-430d-acd1-0b93599407c6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf\" (UID: \"2c30c58b-4263-430d-acd1-0b93599407c6\") " pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf"
Nov 03 20:11:14.591883 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.591847    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjlqv\" (UniqueName: \"kubernetes.io/projected/2c30c58b-4263-430d-acd1-0b93599407c6-kube-api-access-fjlqv\") pod \"payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf\" (UID: \"2c30c58b-4263-430d-acd1-0b93599407c6\") " pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf"
Nov 03 20:11:14.592909 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.592873    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c30c58b-4263-430d-acd1-0b93599407c6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf\" (UID: \"2c30c58b-4263-430d-acd1-0b93599407c6\") " pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf"
Nov 03 20:11:14.593842 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.583104    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c30c58b-4263-430d-acd1-0b93599407c6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf\" (UID: \"2c30c58b-4263-430d-acd1-0b93599407c6\") " pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf"
Nov 03 20:11:14.612886 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.612819    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjlqv\" (UniqueName: \"kubernetes.io/projected/2c30c58b-4263-430d-acd1-0b93599407c6-kube-api-access-fjlqv\") pod \"payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf\" (UID: \"2c30c58b-4263-430d-acd1-0b93599407c6\") " pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf"
Nov 03 20:11:14.623018 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.620757    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c30c58b-4263-430d-acd1-0b93599407c6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf\" (UID: \"2c30c58b-4263-430d-acd1-0b93599407c6\") " pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf"
Nov 03 20:11:14.691241 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:14.691098    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf"
Nov 03 20:11:15.108487 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:11:15.108452    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c30c58b_4263_430d_acd1_0b93599407c6.slice/crio-66244fd2dc3b33d9828a5625712944362f5b46e01974566baf47edbbf3d3cdd1 WatchSource:0}: Error finding container 66244fd2dc3b33d9828a5625712944362f5b46e01974566baf47edbbf3d3cdd1: Status 404 returned error can't find the container with id 66244fd2dc3b33d9828a5625712944362f5b46e01974566baf47edbbf3d3cdd1
Nov 03 20:11:15.108690 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:15.108666    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf"]
Nov 03 20:11:15.462332 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:15.462240    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf" event={"ID":"2c30c58b-4263-430d-acd1-0b93599407c6","Type":"ContainerStarted","Data":"1f07e5b56d46870fd44192d731c59019220bd7a92ed2ca5f8826c6ffe270e890"}
Nov 03 20:11:15.462332 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:15.462302    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf" event={"ID":"2c30c58b-4263-430d-acd1-0b93599407c6","Type":"ContainerStarted","Data":"66244fd2dc3b33d9828a5625712944362f5b46e01974566baf47edbbf3d3cdd1"}
Nov 03 20:11:15.462332 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:15.462323    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf"
Nov 03 20:11:15.482318 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:15.482263    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf" podStartSLOduration=1.482245152 podStartE2EDuration="1.482245152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:15.481146682 +0000 UTC m=+2024.253439725" watchObservedRunningTime="2026-11-03 20:11:15.482245152 +0000 UTC m=+2024.254538168"
Nov 03 20:11:17.895013 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:17.894975    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:11:21.224631 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:21.224593    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7"]
Nov 03 20:11:21.226032 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:21.225987    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7" podUID="3b1f58d2-241a-45ee-8e9c-32def711457b" containerName="maas-api" containerID="cri-o://d532ac953c44a308002a487e9dda01bcf3bf668a9e528ddc606c061ba639b8d3" gracePeriod=30
Nov 03 20:11:21.923835 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:21.923623    5591 generic.go:356] "Generic (PLEG): container finished" podID="3b1f58d2-241a-45ee-8e9c-32def711457b" containerID="d532ac953c44a308002a487e9dda01bcf3bf668a9e528ddc606c061ba639b8d3" exitCode=0
Nov 03 20:11:21.924049 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:21.923890    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7" event={"ID":"3b1f58d2-241a-45ee-8e9c-32def711457b","Type":"ContainerDied","Data":"d532ac953c44a308002a487e9dda01bcf3bf668a9e528ddc606c061ba639b8d3"}
Nov 03 20:11:21.928390 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:21.928357    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7"
Nov 03 20:11:22.102825 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:22.102742    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b1f58d2-241a-45ee-8e9c-32def711457b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1f58d2-241a-45ee-8e9c-32def711457b-metrics-certs\") pod \"3b1f58d2-241a-45ee-8e9c-32def711457b\" (UID: \"3b1f58d2-241a-45ee-8e9c-32def711457b\") "
Nov 03 20:11:22.102825 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:22.102816    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b1f58d2-241a-45ee-8e9c-32def711457b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1f58d2-241a-45ee-8e9c-32def711457b-openshift-service-ca\") pod \"3b1f58d2-241a-45ee-8e9c-32def711457b\" (UID: \"3b1f58d2-241a-45ee-8e9c-32def711457b\") "
Nov 03 20:11:22.103106 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:22.103081    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b1f58d2-241a-45ee-8e9c-32def711457b-kube-api-access-tdbnw\" (UniqueName: \"kubernetes.io/projected/3b1f58d2-241a-45ee-8e9c-32def711457b-kube-api-access-tdbnw\") pod \"3b1f58d2-241a-45ee-8e9c-32def711457b\" (UID: \"3b1f58d2-241a-45ee-8e9c-32def711457b\") "
Nov 03 20:11:22.103106 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:22.103107    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b1f58d2-241a-45ee-8e9c-32def711457b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b1f58d2-241a-45ee-8e9c-32def711457b-maas-api-tls\") pod \"3b1f58d2-241a-45ee-8e9c-32def711457b\" (UID: \"3b1f58d2-241a-45ee-8e9c-32def711457b\") "
Nov 03 20:11:22.103257 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:22.103135    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b1f58d2-241a-45ee-8e9c-32def711457b-openshift-service-ca" pod "3b1f58d2-241a-45ee-8e9c-32def711457b" (UID: "3b1f58d2-241a-45ee-8e9c-32def711457b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:11:22.103727 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:22.103698    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1f58d2-241a-45ee-8e9c-32def711457b-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:22.105193 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:22.105165    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b1f58d2-241a-45ee-8e9c-32def711457b-metrics-certs" pod "3b1f58d2-241a-45ee-8e9c-32def711457b" (UID: "3b1f58d2-241a-45ee-8e9c-32def711457b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:11:22.105370 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:22.105349    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b1f58d2-241a-45ee-8e9c-32def711457b-maas-api-tls" pod "3b1f58d2-241a-45ee-8e9c-32def711457b" (UID: "3b1f58d2-241a-45ee-8e9c-32def711457b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:11:22.105370 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:22.105357    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1f58d2-241a-45ee-8e9c-32def711457b-kube-api-access-tdbnw" pod "3b1f58d2-241a-45ee-8e9c-32def711457b" (UID: "3b1f58d2-241a-45ee-8e9c-32def711457b"). InnerVolumeSpecName "kube-api-access-tdbnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:22.209270 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:22.206708    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1f58d2-241a-45ee-8e9c-32def711457b-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:22.209270 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:22.206753    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdbnw\" (UniqueName: \"kubernetes.io/projected/3b1f58d2-241a-45ee-8e9c-32def711457b-kube-api-access-tdbnw\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:22.209270 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:22.206771    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b1f58d2-241a-45ee-8e9c-32def711457b-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:22.935938 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:22.935908    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7"
Nov 03 20:11:22.936377 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:22.935937    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7" event={"ID":"3b1f58d2-241a-45ee-8e9c-32def711457b","Type":"ContainerDied","Data":"c47651426b8ccd18d7939deb417365e3bdc0087bbec3b6703f00788e353556ad"}
Nov 03 20:11:22.936377 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:22.935999    5591 scope.go:122] "RemoveContainer" containerID="d532ac953c44a308002a487e9dda01bcf3bf668a9e528ddc606c061ba639b8d3"
Nov 03 20:11:22.963522 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:22.963484    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7"]
Nov 03 20:11:22.967684 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:22.967658    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-36556d-7c5c45c69b-r9lr7"]
Nov 03 20:11:23.353499 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:23.353465    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b1f58d2-241a-45ee-8e9c-32def711457b" path="/var/lib/kubelet/pods/3b1f58d2-241a-45ee-8e9c-32def711457b/volumes"
Nov 03 20:11:24.436099 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:24.436055    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658"]
Nov 03 20:11:24.437344 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:24.437317    5591 cpu_manager.go:414] "containerMap: removing container" podUID="3b1f58d2-241a-45ee-8e9c-32def711457b" containerName="maas-api"
Nov 03 20:11:24.437344 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:24.437342    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b1f58d2-241a-45ee-8e9c-32def711457b" containerName="maas-api"
Nov 03 20:11:24.437595 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:24.437581    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b1f58d2-241a-45ee-8e9c-32def711457b" containerName="maas-api"
Nov 03 20:11:24.441798 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:24.441774    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658"
Nov 03 20:11:24.447808 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:24.447746    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"agc-cred-9d0b4d393cba\""
Nov 03 20:11:24.448033 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:24.447848    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"agc-cred-2043ca6290ee\""
Nov 03 20:11:24.448867 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:24.448840    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"kube-root-ca.crt\""
Nov 03 20:11:24.448867 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:24.448857    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"routing-overlay\""
Nov 03 20:11:24.449037 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:24.448862    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"payload-processing-external-model-plugins-e2e-extmod-b-46a6d7\""
Nov 03 20:11:24.449037 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:24.448956    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"openshift-service-ca.crt\""
Nov 03 20:11:24.449037 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:24.448966    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"payload-processing-external-model-e2e--a1bcdb2a-dockercfg-5sh7w\""
Nov 03 20:11:24.451946 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:24.451910    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658"]
Nov 03 20:11:24.490156 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:24.490113    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq6mg\" (UniqueName: \"kubernetes.io/projected/a06f29e1-b0c1-47e7-b121-783c8ce53525-kube-api-access-xq6mg\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658\" (UID: \"a06f29e1-b0c1-47e7-b121-783c8ce53525\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658"
Nov 03 20:11:24.490552 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:24.490525    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a06f29e1-b0c1-47e7-b121-783c8ce53525-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658\" (UID: \"a06f29e1-b0c1-47e7-b121-783c8ce53525\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658"
Nov 03 20:11:24.490659 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:24.490569    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06f29e1-b0c1-47e7-b121-783c8ce53525-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658\" (UID: \"a06f29e1-b0c1-47e7-b121-783c8ce53525\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658"
Nov 03 20:11:24.490723 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:24.490689    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a06f29e1-b0c1-47e7-b121-783c8ce53525-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658\" (UID: \"a06f29e1-b0c1-47e7-b121-783c8ce53525\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658"
Nov 03 20:11:24.667631 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:24.667591    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a06f29e1-b0c1-47e7-b121-783c8ce53525-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658\" (UID: \"a06f29e1-b0c1-47e7-b121-783c8ce53525\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658"
Nov 03 20:11:24.667877 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:24.667648    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06f29e1-b0c1-47e7-b121-783c8ce53525-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658\" (UID: \"a06f29e1-b0c1-47e7-b121-783c8ce53525\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658"
Nov 03 20:11:24.667877 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:24.667744    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a06f29e1-b0c1-47e7-b121-783c8ce53525-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658\" (UID: \"a06f29e1-b0c1-47e7-b121-783c8ce53525\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658"
Nov 03 20:11:24.667877 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:24.667852    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq6mg\" (UniqueName: \"kubernetes.io/projected/a06f29e1-b0c1-47e7-b121-783c8ce53525-kube-api-access-xq6mg\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658\" (UID: \"a06f29e1-b0c1-47e7-b121-783c8ce53525\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658"
Nov 03 20:11:24.668684 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:24.668657    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a06f29e1-b0c1-47e7-b121-783c8ce53525-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658\" (UID: \"a06f29e1-b0c1-47e7-b121-783c8ce53525\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658"
Nov 03 20:11:24.668962 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:24.668939    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06f29e1-b0c1-47e7-b121-783c8ce53525-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658\" (UID: \"a06f29e1-b0c1-47e7-b121-783c8ce53525\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658"
Nov 03 20:11:24.675232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:24.673335    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a06f29e1-b0c1-47e7-b121-783c8ce53525-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658\" (UID: \"a06f29e1-b0c1-47e7-b121-783c8ce53525\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658"
Nov 03 20:11:24.680618 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:24.680589    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq6mg\" (UniqueName: \"kubernetes.io/projected/a06f29e1-b0c1-47e7-b121-783c8ce53525-kube-api-access-xq6mg\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658\" (UID: \"a06f29e1-b0c1-47e7-b121-783c8ce53525\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658"
Nov 03 20:11:24.764319 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:24.764287    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658"
Nov 03 20:11:25.060072 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:25.060039    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658"]
Nov 03 20:11:25.060194 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:11:25.060085    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06f29e1_b0c1_47e7_b121_783c8ce53525.slice/crio-9d636bba134ff5bb38b15f23e1d32a9f0d237fc8dd44d22e51835eee8657d248 WatchSource:0}: Error finding container 9d636bba134ff5bb38b15f23e1d32a9f0d237fc8dd44d22e51835eee8657d248: Status 404 returned error can't find the container with id 9d636bba134ff5bb38b15f23e1d32a9f0d237fc8dd44d22e51835eee8657d248
Nov 03 20:11:25.968793 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:25.968746    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658" event={"ID":"a06f29e1-b0c1-47e7-b121-783c8ce53525","Type":"ContainerStarted","Data":"b40159a058ff67021491e6901b6a0c131ee9b6c1abc9ac0b04ebc133084c8ebf"}
Nov 03 20:11:25.969275 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:25.968819    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658" event={"ID":"a06f29e1-b0c1-47e7-b121-783c8ce53525","Type":"ContainerStarted","Data":"9d636bba134ff5bb38b15f23e1d32a9f0d237fc8dd44d22e51835eee8657d248"}
Nov 03 20:11:25.969275 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:25.968847    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658"
Nov 03 20:11:25.989758 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:25.989691    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658" podStartSLOduration=1.989669293 podStartE2EDuration="1.989669293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:25.989591903 +0000 UTC m=+2034.761884932" watchObservedRunningTime="2026-11-03 20:11:25.989669293 +0000 UTC m=+2034.761962311"
Nov 03 20:11:26.474395 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:26.474363    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf"
Nov 03 20:11:28.728482 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:11:28.728440    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:11:29.913029 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:29.912987    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"]
Nov 03 20:11:29.913688 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:29.913336    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn" podUID="740ca9b8-631f-4381-b864-22b201e29d11" containerName="istio-proxy" containerID="cri-o://0800e396985cee475c4e8982d74125b954f8bb4511c4b0f5ff071df2e9e092de" gracePeriod=30
Nov 03 20:11:29.929272 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:29.929233    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"]
Nov 03 20:11:29.929548 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:29.929513    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk" podUID="e2d939a4-ec3b-4a1a-8060-cc73381fc711" containerName="istio-proxy" containerID="cri-o://f71692c47e3066cfe6f1ca749e1bce6fef38a4d872ac9f0a1d849c954a9569fb" gracePeriod=30
Nov 03 20:11:35.111042 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:11:35.110948    5591 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774a0a2c_2534_4a5c_9c2d_737b3aef7268.slice/crio-6bbdc22aef988ece39856be4219066b553a875fc9924cbdf7a76b28e2b65a1df\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d939a4_ec3b_4a1a_8060_cc73381fc711.slice/crio-9324704bc7b22c4dd1bed02e78097a4c5ef5606628176c9c0a1c6b0bdcf60e25\": RecentStats: unable to find data in memory cache]"
Nov 03 20:11:35.183108 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.183004    5591 generic.go:356] "Generic (PLEG): container finished" podID="740ca9b8-631f-4381-b864-22b201e29d11" containerID="0800e396985cee475c4e8982d74125b954f8bb4511c4b0f5ff071df2e9e092de" exitCode=0
Nov 03 20:11:35.183108 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.183080    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn" event={"ID":"740ca9b8-631f-4381-b864-22b201e29d11","Type":"ContainerDied","Data":"0800e396985cee475c4e8982d74125b954f8bb4511c4b0f5ff071df2e9e092de"}
Nov 03 20:11:35.185076 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.185048    5591 generic.go:356] "Generic (PLEG): container finished" podID="e2d939a4-ec3b-4a1a-8060-cc73381fc711" containerID="f71692c47e3066cfe6f1ca749e1bce6fef38a4d872ac9f0a1d849c954a9569fb" exitCode=0
Nov 03 20:11:35.185229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.185125    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk" event={"ID":"e2d939a4-ec3b-4a1a-8060-cc73381fc711","Type":"ContainerDied","Data":"f71692c47e3066cfe6f1ca749e1bce6fef38a4d872ac9f0a1d849c954a9569fb"}
Nov 03 20:11:35.331988 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.331938    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:11:35.383265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.383188    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:11:35.404679 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.404641    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-istio-envoy\") pod \"740ca9b8-631f-4381-b864-22b201e29d11\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") "
Nov 03 20:11:35.404873 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.404706    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/740ca9b8-631f-4381-b864-22b201e29d11-istio-token\" (UniqueName: \"kubernetes.io/projected/740ca9b8-631f-4381-b864-22b201e29d11-istio-token\") pod \"740ca9b8-631f-4381-b864-22b201e29d11\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") "
Nov 03 20:11:35.404873 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.404731    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-istio-data\") pod \"740ca9b8-631f-4381-b864-22b201e29d11\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") "
Nov 03 20:11:35.404873 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.404822    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-workload-certs\") pod \"740ca9b8-631f-4381-b864-22b201e29d11\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") "
Nov 03 20:11:35.404873 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.404853    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/740ca9b8-631f-4381-b864-22b201e29d11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/740ca9b8-631f-4381-b864-22b201e29d11-istio-podinfo\") pod \"740ca9b8-631f-4381-b864-22b201e29d11\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") "
Nov 03 20:11:35.405120 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.404999    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/740ca9b8-631f-4381-b864-22b201e29d11-kube-api-access-54l5l\" (UniqueName: \"kubernetes.io/projected/740ca9b8-631f-4381-b864-22b201e29d11-kube-api-access-54l5l\") pod \"740ca9b8-631f-4381-b864-22b201e29d11\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") "
Nov 03 20:11:35.405120 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.405057    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/740ca9b8-631f-4381-b864-22b201e29d11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/740ca9b8-631f-4381-b864-22b201e29d11-istiod-ca-cert\") pod \"740ca9b8-631f-4381-b864-22b201e29d11\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") "
Nov 03 20:11:35.405350 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.405136    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-istio-data" pod "740ca9b8-631f-4381-b864-22b201e29d11" (UID: "740ca9b8-631f-4381-b864-22b201e29d11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:35.405350 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.405155    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-workload-certs" pod "740ca9b8-631f-4381-b864-22b201e29d11" (UID: "740ca9b8-631f-4381-b864-22b201e29d11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:35.405350 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.405159    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-workload-socket\") pod \"740ca9b8-631f-4381-b864-22b201e29d11\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") "
Nov 03 20:11:35.405350 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.405264    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-credential-socket\") pod \"740ca9b8-631f-4381-b864-22b201e29d11\" (UID: \"740ca9b8-631f-4381-b864-22b201e29d11\") "
Nov 03 20:11:35.405612 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.405374    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-workload-socket" pod "740ca9b8-631f-4381-b864-22b201e29d11" (UID: "740ca9b8-631f-4381-b864-22b201e29d11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:35.405612 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.405485    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/740ca9b8-631f-4381-b864-22b201e29d11-istiod-ca-cert" pod "740ca9b8-631f-4381-b864-22b201e29d11" (UID: "740ca9b8-631f-4381-b864-22b201e29d11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:11:35.405612 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.405570    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-credential-socket" pod "740ca9b8-631f-4381-b864-22b201e29d11" (UID: "740ca9b8-631f-4381-b864-22b201e29d11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:35.406620 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.406225    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.406620 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.406250    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.406620 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.406307    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.406620 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.406421    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.406620 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.406435    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/740ca9b8-631f-4381-b864-22b201e29d11-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.408979 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.408946    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/740ca9b8-631f-4381-b864-22b201e29d11-kube-api-access-54l5l" pod "740ca9b8-631f-4381-b864-22b201e29d11" (UID: "740ca9b8-631f-4381-b864-22b201e29d11"). InnerVolumeSpecName "kube-api-access-54l5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:35.409413 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.409383    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-istio-envoy" pod "740ca9b8-631f-4381-b864-22b201e29d11" (UID: "740ca9b8-631f-4381-b864-22b201e29d11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:35.409527 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.409467    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/740ca9b8-631f-4381-b864-22b201e29d11-istio-podinfo" pod "740ca9b8-631f-4381-b864-22b201e29d11" (UID: "740ca9b8-631f-4381-b864-22b201e29d11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:11:35.411103 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.410810    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/740ca9b8-631f-4381-b864-22b201e29d11-istio-token" pod "740ca9b8-631f-4381-b864-22b201e29d11" (UID: "740ca9b8-631f-4381-b864-22b201e29d11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:35.508095 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.507992    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-workload-certs\") pod \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") "
Nov 03 20:11:35.508350 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.508124    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-workload-socket\") pod \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") "
Nov 03 20:11:35.508418 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.508392    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-envoy\") pod \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") "
Nov 03 20:11:35.508491 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.508473    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-podinfo\") pod \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") "
Nov 03 20:11:35.508569 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.508553    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-credential-socket\") pod \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") "
Nov 03 20:11:35.508634 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.508590    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-data\") pod \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") "
Nov 03 20:11:35.508744 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.508728    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istiod-ca-cert\") pod \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") "
Nov 03 20:11:35.508801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.508760    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2d939a4-ec3b-4a1a-8060-cc73381fc711-kube-api-access-4824q\" (UniqueName: \"kubernetes.io/projected/e2d939a4-ec3b-4a1a-8060-cc73381fc711-kube-api-access-4824q\") pod \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") "
Nov 03 20:11:35.508860 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.508805    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-token\" (UniqueName: \"kubernetes.io/projected/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-token\") pod \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\" (UID: \"e2d939a4-ec3b-4a1a-8060-cc73381fc711\") "
Nov 03 20:11:35.509841 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.509699    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/740ca9b8-631f-4381-b864-22b201e29d11-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.509841 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.509725    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/740ca9b8-631f-4381-b864-22b201e29d11-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.509841 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.509740    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/740ca9b8-631f-4381-b864-22b201e29d11-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.509841 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.509755    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54l5l\" (UniqueName: \"kubernetes.io/projected/740ca9b8-631f-4381-b864-22b201e29d11-kube-api-access-54l5l\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.512843 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.512644    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-podinfo" pod "e2d939a4-ec3b-4a1a-8060-cc73381fc711" (UID: "e2d939a4-ec3b-4a1a-8060-cc73381fc711"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:11:35.512843 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.512765    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-token" pod "e2d939a4-ec3b-4a1a-8060-cc73381fc711" (UID: "e2d939a4-ec3b-4a1a-8060-cc73381fc711"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:35.513032 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.512999    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-credential-socket" pod "e2d939a4-ec3b-4a1a-8060-cc73381fc711" (UID: "e2d939a4-ec3b-4a1a-8060-cc73381fc711"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:35.513112 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.513036    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-workload-certs" pod "e2d939a4-ec3b-4a1a-8060-cc73381fc711" (UID: "e2d939a4-ec3b-4a1a-8060-cc73381fc711"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:35.513385 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.513296    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-data" pod "e2d939a4-ec3b-4a1a-8060-cc73381fc711" (UID: "e2d939a4-ec3b-4a1a-8060-cc73381fc711"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:35.513385 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.513315    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-workload-socket" pod "e2d939a4-ec3b-4a1a-8060-cc73381fc711" (UID: "e2d939a4-ec3b-4a1a-8060-cc73381fc711"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:35.513831 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.513791    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istiod-ca-cert" pod "e2d939a4-ec3b-4a1a-8060-cc73381fc711" (UID: "e2d939a4-ec3b-4a1a-8060-cc73381fc711"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:11:35.515948 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.515925    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-envoy" pod "e2d939a4-ec3b-4a1a-8060-cc73381fc711" (UID: "e2d939a4-ec3b-4a1a-8060-cc73381fc711"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:11:35.515948 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.515942    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d939a4-ec3b-4a1a-8060-cc73381fc711-kube-api-access-4824q" pod "e2d939a4-ec3b-4a1a-8060-cc73381fc711" (UID: "e2d939a4-ec3b-4a1a-8060-cc73381fc711"). InnerVolumeSpecName "kube-api-access-4824q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:11:35.611489 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.611448    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.611489 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.611491    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4824q\" (UniqueName: \"kubernetes.io/projected/e2d939a4-ec3b-4a1a-8060-cc73381fc711-kube-api-access-4824q\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.611757 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.611505    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.611757 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.611516    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.611757 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.611525    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.611757 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.611533    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.611757 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.611542    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.611757 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.611552    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:35.611757 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:35.611560    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2d939a4-ec3b-4a1a-8060-cc73381fc711-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:11:36.233101 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:36.233028    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"
Nov 03 20:11:36.233574 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:36.233017    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn" event={"ID":"740ca9b8-631f-4381-b864-22b201e29d11","Type":"ContainerDied","Data":"a93dac37e039405f21a9097ee47ffd3447350d6449f3777d22b496cdb5c9ac39"}
Nov 03 20:11:36.233574 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:36.233255    5591 scope.go:122] "RemoveContainer" containerID="0800e396985cee475c4e8982d74125b954f8bb4511c4b0f5ff071df2e9e092de"
Nov 03 20:11:36.241098 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:36.241056    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk" event={"ID":"e2d939a4-ec3b-4a1a-8060-cc73381fc711","Type":"ContainerDied","Data":"9324704bc7b22c4dd1bed02e78097a4c5ef5606628176c9c0a1c6b0bdcf60e25"}
Nov 03 20:11:36.241302 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:36.241284    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"
Nov 03 20:11:36.250042 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:36.250003    5591 scope.go:122] "RemoveContainer" containerID="f71692c47e3066cfe6f1ca749e1bce6fef38a4d872ac9f0a1d849c954a9569fb"
Nov 03 20:11:36.268908 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:36.268298    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"]
Nov 03 20:11:36.274254 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:36.274225    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-5n8rn"]
Nov 03 20:11:36.283080 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:36.283018    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"]
Nov 03 20:11:36.293159 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:36.293124    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-36556d-openshift-default-658cbc6465-nmhgk"]
Nov 03 20:11:36.982455 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:36.982419    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658"
Nov 03 20:11:37.648250 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:37.647344    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="740ca9b8-631f-4381-b864-22b201e29d11" path="/var/lib/kubelet/pods/740ca9b8-631f-4381-b864-22b201e29d11/volumes"
Nov 03 20:11:37.648923 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:37.648259    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2d939a4-ec3b-4a1a-8060-cc73381fc711" path="/var/lib/kubelet/pods/e2d939a4-ec3b-4a1a-8060-cc73381fc711/volumes"
Nov 03 20:11:54.845603 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:54.845563    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6"]
Nov 03 20:11:54.847033 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:54.847004    5591 cpu_manager.go:414] "containerMap: removing container" podUID="740ca9b8-631f-4381-b864-22b201e29d11" containerName="istio-proxy"
Nov 03 20:11:54.847033 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:54.847031    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="740ca9b8-631f-4381-b864-22b201e29d11" containerName="istio-proxy"
Nov 03 20:11:54.847248 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:54.847134    5591 cpu_manager.go:414] "containerMap: removing container" podUID="e2d939a4-ec3b-4a1a-8060-cc73381fc711" containerName="istio-proxy"
Nov 03 20:11:54.847248 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:54.847143    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2d939a4-ec3b-4a1a-8060-cc73381fc711" containerName="istio-proxy"
Nov 03 20:11:54.847413 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:54.847396    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2d939a4-ec3b-4a1a-8060-cc73381fc711" containerName="istio-proxy"
Nov 03 20:11:54.847472 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:54.847425    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="740ca9b8-631f-4381-b864-22b201e29d11" containerName="istio-proxy"
Nov 03 20:11:54.851654 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:54.851632    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6"
Nov 03 20:11:54.858752 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:54.858724    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-33e267-serving-cert\""
Nov 03 20:11:54.868326 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:54.868301    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6"]
Nov 03 20:11:54.952234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:54.952156    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqhrf\" (UniqueName: \"kubernetes.io/projected/92c8f021-930c-438f-baf2-7457458d03af-kube-api-access-tqhrf\") pod \"maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6\" (UID: \"92c8f021-930c-438f-baf2-7457458d03af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6"
Nov 03 20:11:54.952435 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:54.952244    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92c8f021-930c-438f-baf2-7457458d03af-maas-api-tls\") pod \"maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6\" (UID: \"92c8f021-930c-438f-baf2-7457458d03af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6"
Nov 03 20:11:54.952435 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:54.952344    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c8f021-930c-438f-baf2-7457458d03af-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6\" (UID: \"92c8f021-930c-438f-baf2-7457458d03af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6"
Nov 03 20:11:54.952558 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:54.952535    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92c8f021-930c-438f-baf2-7457458d03af-metrics-certs\") pod \"maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6\" (UID: \"92c8f021-930c-438f-baf2-7457458d03af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6"
Nov 03 20:11:55.061748 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:55.061697    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c8f021-930c-438f-baf2-7457458d03af-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6\" (UID: \"92c8f021-930c-438f-baf2-7457458d03af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6"
Nov 03 20:11:55.067573 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:55.067527    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92c8f021-930c-438f-baf2-7457458d03af-metrics-certs\") pod \"maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6\" (UID: \"92c8f021-930c-438f-baf2-7457458d03af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6"
Nov 03 20:11:55.080786 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:55.068101    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqhrf\" (UniqueName: \"kubernetes.io/projected/92c8f021-930c-438f-baf2-7457458d03af-kube-api-access-tqhrf\") pod \"maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6\" (UID: \"92c8f021-930c-438f-baf2-7457458d03af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6"
Nov 03 20:11:55.081792 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:55.081761    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92c8f021-930c-438f-baf2-7457458d03af-maas-api-tls\") pod \"maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6\" (UID: \"92c8f021-930c-438f-baf2-7457458d03af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6"
Nov 03 20:11:55.082660 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:55.063095    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c8f021-930c-438f-baf2-7457458d03af-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6\" (UID: \"92c8f021-930c-438f-baf2-7457458d03af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6"
Nov 03 20:11:55.106627 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:55.104928    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqhrf\" (UniqueName: \"kubernetes.io/projected/92c8f021-930c-438f-baf2-7457458d03af-kube-api-access-tqhrf\") pod \"maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6\" (UID: \"92c8f021-930c-438f-baf2-7457458d03af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6"
Nov 03 20:11:55.106860 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:55.106803    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92c8f021-930c-438f-baf2-7457458d03af-metrics-certs\") pod \"maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6\" (UID: \"92c8f021-930c-438f-baf2-7457458d03af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6"
Nov 03 20:11:55.111345 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:55.110922    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92c8f021-930c-438f-baf2-7457458d03af-maas-api-tls\") pod \"maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6\" (UID: \"92c8f021-930c-438f-baf2-7457458d03af\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6"
Nov 03 20:11:55.174835 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:55.174796    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6"
Nov 03 20:11:55.585003 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:55.584953    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6"]
Nov 03 20:11:55.625735 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:55.625637    5591 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:11:56.639581 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:56.639536    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6" event={"ID":"92c8f021-930c-438f-baf2-7457458d03af","Type":"ContainerStarted","Data":"1313bdce7771ec9e4c62672e9bc2960add3a9ee9b78d5f7f6ab75a8ac080322e"}
Nov 03 20:11:56.640239 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:56.639617    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6" event={"ID":"92c8f021-930c-438f-baf2-7457458d03af","Type":"ContainerStarted","Data":"fb58a7271b2bcd087dd5a8f3f75a7ae5da351c2f5c67d1499f764301b59f24c0"}
Nov 03 20:11:56.640239 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:56.639777    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6"
Nov 03 20:11:56.687227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:11:56.686424    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6" podStartSLOduration=2.186608352 podStartE2EDuration="2.686399372s" totalImagesPullingTime="499.79102ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:56.671245863 +0000 UTC m=+2065.443538881" watchObservedRunningTime="2026-11-03 20:11:56.686399372 +0000 UTC m=+2065.458692391"
Nov 03 20:11:58.729373 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:11:58.729324    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:12:02.669855 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:02.669815    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6"
Nov 03 20:12:12.436118 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:12.436014    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:12:18.920231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:18.917292    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg"]
Nov 03 20:12:19.046234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:19.045772    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg"]
Nov 03 20:12:19.046234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:19.045893    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg"
Nov 03 20:12:19.057915 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:19.057881    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0e4ce7\"/\"agc-cred-4cbea2d0a3c6\""
Nov 03 20:12:19.203637 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:19.203539    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/330b5625-e53f-4c73-93cb-98889bc2f0b8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg\" (UID: \"330b5625-e53f-4c73-93cb-98889bc2f0b8\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg"
Nov 03 20:12:19.203637 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:19.203611    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gst8n\" (UniqueName: \"kubernetes.io/projected/330b5625-e53f-4c73-93cb-98889bc2f0b8-kube-api-access-gst8n\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg\" (UID: \"330b5625-e53f-4c73-93cb-98889bc2f0b8\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg"
Nov 03 20:12:19.203920 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:19.203900    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/330b5625-e53f-4c73-93cb-98889bc2f0b8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg\" (UID: \"330b5625-e53f-4c73-93cb-98889bc2f0b8\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg"
Nov 03 20:12:19.204111 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:19.204084    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/330b5625-e53f-4c73-93cb-98889bc2f0b8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg\" (UID: \"330b5625-e53f-4c73-93cb-98889bc2f0b8\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg"
Nov 03 20:12:19.305350 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:19.305320    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/330b5625-e53f-4c73-93cb-98889bc2f0b8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg\" (UID: \"330b5625-e53f-4c73-93cb-98889bc2f0b8\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg"
Nov 03 20:12:19.305566 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:19.305375    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gst8n\" (UniqueName: \"kubernetes.io/projected/330b5625-e53f-4c73-93cb-98889bc2f0b8-kube-api-access-gst8n\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg\" (UID: \"330b5625-e53f-4c73-93cb-98889bc2f0b8\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg"
Nov 03 20:12:19.305718 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:19.305568    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/330b5625-e53f-4c73-93cb-98889bc2f0b8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg\" (UID: \"330b5625-e53f-4c73-93cb-98889bc2f0b8\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg"
Nov 03 20:12:19.305718 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:19.305658    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/330b5625-e53f-4c73-93cb-98889bc2f0b8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg\" (UID: \"330b5625-e53f-4c73-93cb-98889bc2f0b8\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg"
Nov 03 20:12:19.306477 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:19.306454    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/330b5625-e53f-4c73-93cb-98889bc2f0b8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg\" (UID: \"330b5625-e53f-4c73-93cb-98889bc2f0b8\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg"
Nov 03 20:12:19.306585 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:19.306506    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/330b5625-e53f-4c73-93cb-98889bc2f0b8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg\" (UID: \"330b5625-e53f-4c73-93cb-98889bc2f0b8\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg"
Nov 03 20:12:19.308356 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:19.308329    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/330b5625-e53f-4c73-93cb-98889bc2f0b8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg\" (UID: \"330b5625-e53f-4c73-93cb-98889bc2f0b8\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg"
Nov 03 20:12:19.330355 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:19.330325    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gst8n\" (UniqueName: \"kubernetes.io/projected/330b5625-e53f-4c73-93cb-98889bc2f0b8-kube-api-access-gst8n\") pod \"payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg\" (UID: \"330b5625-e53f-4c73-93cb-98889bc2f0b8\") " pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg"
Nov 03 20:12:19.417773 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:19.417433    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg"
Nov 03 20:12:19.719035 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:19.719003    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg"]
Nov 03 20:12:19.725049 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:12:19.725014    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330b5625_e53f_4c73_93cb_98889bc2f0b8.slice/crio-3292c1715b74bfd7b52729d0f5bab59a94349368b982111f570c06c7337904b1 WatchSource:0}: Error finding container 3292c1715b74bfd7b52729d0f5bab59a94349368b982111f570c06c7337904b1: Status 404 returned error can't find the container with id 3292c1715b74bfd7b52729d0f5bab59a94349368b982111f570c06c7337904b1
Nov 03 20:12:20.117683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:20.117648    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg" event={"ID":"330b5625-e53f-4c73-93cb-98889bc2f0b8","Type":"ContainerStarted","Data":"9896e99c9600cfd9376ed843627e7aad02cc341a3d12a4764abe126eed2e7162"}
Nov 03 20:12:20.118104 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:20.117703    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg" event={"ID":"330b5625-e53f-4c73-93cb-98889bc2f0b8","Type":"ContainerStarted","Data":"3292c1715b74bfd7b52729d0f5bab59a94349368b982111f570c06c7337904b1"}
Nov 03 20:12:20.118104 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:20.117868    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg"
Nov 03 20:12:20.138612 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:20.138548    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg" podStartSLOduration=2.138532725 podStartE2EDuration="2.138532725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:20.136657994 +0000 UTC m=+2088.908951021" watchObservedRunningTime="2026-11-03 20:12:20.138532725 +0000 UTC m=+2088.910825742"
Nov 03 20:12:25.993702 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:25.993667    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf"]
Nov 03 20:12:25.999604 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:25.999578    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf"
Nov 03 20:12:26.008060 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:26.007103    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-4c2a28\""
Nov 03 20:12:26.008060 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:26.007196    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-4c2a28-dockercfg-psjnh\""
Nov 03 20:12:26.013379 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:26.013335    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf"]
Nov 03 20:12:26.214230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:26.204513    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbkqb\" (UniqueName: \"kubernetes.io/projected/080839df-c38a-4bfc-8b72-008a2c12de63-kube-api-access-wbkqb\") pod \"payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf\" (UID: \"080839df-c38a-4bfc-8b72-008a2c12de63\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf"
Nov 03 20:12:26.214230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:26.204774    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/080839df-c38a-4bfc-8b72-008a2c12de63-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf\" (UID: \"080839df-c38a-4bfc-8b72-008a2c12de63\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf"
Nov 03 20:12:26.307642 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:26.307177    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbkqb\" (UniqueName: \"kubernetes.io/projected/080839df-c38a-4bfc-8b72-008a2c12de63-kube-api-access-wbkqb\") pod \"payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf\" (UID: \"080839df-c38a-4bfc-8b72-008a2c12de63\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf"
Nov 03 20:12:26.307642 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:26.307409    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/080839df-c38a-4bfc-8b72-008a2c12de63-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf\" (UID: \"080839df-c38a-4bfc-8b72-008a2c12de63\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf"
Nov 03 20:12:26.309712 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:26.308419    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/080839df-c38a-4bfc-8b72-008a2c12de63-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf\" (UID: \"080839df-c38a-4bfc-8b72-008a2c12de63\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf"
Nov 03 20:12:26.349684 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:26.349650    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbkqb\" (UniqueName: \"kubernetes.io/projected/080839df-c38a-4bfc-8b72-008a2c12de63-kube-api-access-wbkqb\") pod \"payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf\" (UID: \"080839df-c38a-4bfc-8b72-008a2c12de63\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf"
Nov 03 20:12:26.380298 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:26.380259    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf"
Nov 03 20:12:26.679086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:26.679057    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf"]
Nov 03 20:12:26.679419 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:12:26.679393    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080839df_c38a_4bfc_8b72_008a2c12de63.slice/crio-85b279f1e823aedc37df6a27b43663ea81654375d18658960b997e2c492af2dd WatchSource:0}: Error finding container 85b279f1e823aedc37df6a27b43663ea81654375d18658960b997e2c492af2dd: Status 404 returned error can't find the container with id 85b279f1e823aedc37df6a27b43663ea81654375d18658960b997e2c492af2dd
Nov 03 20:12:27.375836 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:27.375796    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf" event={"ID":"080839df-c38a-4bfc-8b72-008a2c12de63","Type":"ContainerStarted","Data":"bcebf22a6126cceaadded0be8c2e3f0417bb6bb1aba21d966d644853a34ea0cf"}
Nov 03 20:12:27.376335 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:27.375858    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf" event={"ID":"080839df-c38a-4bfc-8b72-008a2c12de63","Type":"ContainerStarted","Data":"85b279f1e823aedc37df6a27b43663ea81654375d18658960b997e2c492af2dd"}
Nov 03 20:12:27.376335 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:27.375925    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf"
Nov 03 20:12:27.398911 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:27.398857    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf" podStartSLOduration=2.398841719 podStartE2EDuration="2.398841719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:27.39446454 +0000 UTC m=+2096.166757561" watchObservedRunningTime="2026-11-03 20:12:27.398841719 +0000 UTC m=+2096.171134737"
Nov 03 20:12:28.726472 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:12:28.726433    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:12:31.129197 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:31.129167    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg"
Nov 03 20:12:31.184036 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:31.184000    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz"]
Nov 03 20:12:31.272402 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:31.272368    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658"]
Nov 03 20:12:32.994476 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:32.994438    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"]
Nov 03 20:12:33.000881 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.000857    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.010815 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.010788    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"]
Nov 03 20:12:33.012554 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.012268    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-8b7351-openshift-default-dockercfg-lx6hb\""
Nov 03 20:12:33.096680 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.096645    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-podinfo\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.096835 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.096709    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-workload-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.096900 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.096843    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-token\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.096900 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.096881    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-envoy\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.096974 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.096948    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-credential-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.097014 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.096988    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjg29\" (UniqueName: \"kubernetes.io/projected/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-kube-api-access-mjg29\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.097157 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.097144    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-workload-certs\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.097241 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.097195    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istiod-ca-cert\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.097288 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.097242    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-data\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.198548 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.198514    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-data\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.198765 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.198626    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-podinfo\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.198765 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.198659    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-workload-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.198765 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.198722    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-token\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.198765 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.198749    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-envoy\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.199062 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.198772    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-credential-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.199062 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.198799    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjg29\" (UniqueName: \"kubernetes.io/projected/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-kube-api-access-mjg29\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.199062 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.198934    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-workload-certs\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.199062 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.199002    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istiod-ca-cert\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.199062 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.199016    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-data\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.199339 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.199266    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-credential-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.199339 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.199298    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-workload-socket\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.199789 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.199351    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-workload-certs\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.199789 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.199746    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istiod-ca-cert\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.201472 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.201439    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-envoy\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.201593 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.201565    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-podinfo\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.213278 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.213255    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-token\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.213619 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.213600    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjg29\" (UniqueName: \"kubernetes.io/projected/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-kube-api-access-mjg29\") pod \"e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") " pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.321339 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.321304    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:33.627387 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:12:33.627353    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4a5d499_e4e8_4489_bbef_bc6e6ffd8b15.slice/crio-54e14ad51654fd3019d5df5b67b4b9641a584f06c82098af2194a53d2d0dc643 WatchSource:0}: Error finding container 54e14ad51654fd3019d5df5b67b4b9641a584f06c82098af2194a53d2d0dc643: Status 404 returned error can't find the container with id 54e14ad51654fd3019d5df5b67b4b9641a584f06c82098af2194a53d2d0dc643
Nov 03 20:12:33.627567 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.627539    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"]
Nov 03 20:12:33.630449 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.630410    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:33.630543 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.630499    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:33.630611 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.630544    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:33.649570 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.649547    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 20:12:33.651074 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:33.651055    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 20:12:34.605874 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:34.605819    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds" event={"ID":"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15","Type":"ContainerStarted","Data":"e4336af13fb5abc05d4709ed8396b95ad3168b3d0662e4d414fcfdfe5ee00913"}
Nov 03 20:12:34.606444 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:34.605888    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds" event={"ID":"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15","Type":"ContainerStarted","Data":"54e14ad51654fd3019d5df5b67b4b9641a584f06c82098af2194a53d2d0dc643"}
Nov 03 20:12:34.634061 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:34.633997    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds" podStartSLOduration=2.633978845 podStartE2EDuration="2.633978845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:34.629814388 +0000 UTC m=+2103.402107432" watchObservedRunningTime="2026-11-03 20:12:34.633978845 +0000 UTC m=+2103.406271866"
Nov 03 20:12:35.322134 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:35.322095    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:35.323909 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:35.323878    5591 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 03 20:12:35.324062 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:35.323935    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds" podUID="e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 03 20:12:36.201276 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.198482    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz" podUID="774a0a2c-2534-4a5c-9c2d-737b3aef7268" containerName="payload-processing" containerID="cri-o://092b1191e6d4c34ad8fc0c0e892c2b6a6358097fdf82913d1ab40599b5094a6c" gracePeriod=25
Nov 03 20:12:36.273661 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.273627    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658" podUID="a06f29e1-b0c1-47e7-b121-783c8ce53525" containerName="payload-processing" containerID="cri-o://b40159a058ff67021491e6901b6a0c131ee9b6c1abc9ac0b04ebc133084c8ebf" gracePeriod=25
Nov 03 20:12:36.321697 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.321666    5591 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 03 20:12:36.321835 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.321730    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds" podUID="e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 03 20:12:36.616542 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.614162    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz"
Nov 03 20:12:36.630802 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.630766    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/774a0a2c-2534-4a5c-9c2d-737b3aef7268-kube-api-access-rdk55\" (UniqueName: \"kubernetes.io/projected/774a0a2c-2534-4a5c-9c2d-737b3aef7268-kube-api-access-rdk55\") pod \"774a0a2c-2534-4a5c-9c2d-737b3aef7268\" (UID: \"774a0a2c-2534-4a5c-9c2d-737b3aef7268\") "
Nov 03 20:12:36.631055 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.631038    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/774a0a2c-2534-4a5c-9c2d-737b3aef7268-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/774a0a2c-2534-4a5c-9c2d-737b3aef7268-plugins-config-volume\") pod \"774a0a2c-2534-4a5c-9c2d-737b3aef7268\" (UID: \"774a0a2c-2534-4a5c-9c2d-737b3aef7268\") "
Nov 03 20:12:36.631179 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.631166    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/774a0a2c-2534-4a5c-9c2d-737b3aef7268-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/774a0a2c-2534-4a5c-9c2d-737b3aef7268-routing-overlay-volume\") pod \"774a0a2c-2534-4a5c-9c2d-737b3aef7268\" (UID: \"774a0a2c-2534-4a5c-9c2d-737b3aef7268\") "
Nov 03 20:12:36.631333 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.631317    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/774a0a2c-2534-4a5c-9c2d-737b3aef7268-provider-credentials\" (UniqueName: \"kubernetes.io/projected/774a0a2c-2534-4a5c-9c2d-737b3aef7268-provider-credentials\") pod \"774a0a2c-2534-4a5c-9c2d-737b3aef7268\" (UID: \"774a0a2c-2534-4a5c-9c2d-737b3aef7268\") "
Nov 03 20:12:36.632223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.632180    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/774a0a2c-2534-4a5c-9c2d-737b3aef7268-plugins-config-volume" pod "774a0a2c-2534-4a5c-9c2d-737b3aef7268" (UID: "774a0a2c-2534-4a5c-9c2d-737b3aef7268"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:36.632646 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.632624    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/774a0a2c-2534-4a5c-9c2d-737b3aef7268-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:36.633083 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.633059    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/774a0a2c-2534-4a5c-9c2d-737b3aef7268-routing-overlay-volume" pod "774a0a2c-2534-4a5c-9c2d-737b3aef7268" (UID: "774a0a2c-2534-4a5c-9c2d-737b3aef7268"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:36.635391 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.635373    5591 generic.go:356] "Generic (PLEG): container finished" podID="a06f29e1-b0c1-47e7-b121-783c8ce53525" containerID="b40159a058ff67021491e6901b6a0c131ee9b6c1abc9ac0b04ebc133084c8ebf" exitCode=0
Nov 03 20:12:36.635613 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.635503    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658" event={"ID":"a06f29e1-b0c1-47e7-b121-783c8ce53525","Type":"ContainerDied","Data":"b40159a058ff67021491e6901b6a0c131ee9b6c1abc9ac0b04ebc133084c8ebf"}
Nov 03 20:12:36.636869 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.636842    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774a0a2c-2534-4a5c-9c2d-737b3aef7268-kube-api-access-rdk55" pod "774a0a2c-2534-4a5c-9c2d-737b3aef7268" (UID: "774a0a2c-2534-4a5c-9c2d-737b3aef7268"). InnerVolumeSpecName "kube-api-access-rdk55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:36.638510 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.637467    5591 generic.go:356] "Generic (PLEG): container finished" podID="774a0a2c-2534-4a5c-9c2d-737b3aef7268" containerID="092b1191e6d4c34ad8fc0c0e892c2b6a6358097fdf82913d1ab40599b5094a6c" exitCode=0
Nov 03 20:12:36.638510 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.638183    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz"
Nov 03 20:12:36.638510 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.638356    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz" event={"ID":"774a0a2c-2534-4a5c-9c2d-737b3aef7268","Type":"ContainerDied","Data":"092b1191e6d4c34ad8fc0c0e892c2b6a6358097fdf82913d1ab40599b5094a6c"}
Nov 03 20:12:36.638510 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.638392    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz" event={"ID":"774a0a2c-2534-4a5c-9c2d-737b3aef7268","Type":"ContainerDied","Data":"6bbdc22aef988ece39856be4219066b553a875fc9924cbdf7a76b28e2b65a1df"}
Nov 03 20:12:36.638510 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.638413    5591 scope.go:122] "RemoveContainer" containerID="092b1191e6d4c34ad8fc0c0e892c2b6a6358097fdf82913d1ab40599b5094a6c"
Nov 03 20:12:36.646272 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.639333    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774a0a2c-2534-4a5c-9c2d-737b3aef7268-provider-credentials" pod "774a0a2c-2534-4a5c-9c2d-737b3aef7268" (UID: "774a0a2c-2534-4a5c-9c2d-737b3aef7268"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:36.668986 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.668960    5591 scope.go:122] "RemoveContainer" containerID="092b1191e6d4c34ad8fc0c0e892c2b6a6358097fdf82913d1ab40599b5094a6c"
Nov 03 20:12:36.669461 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:12:36.669380    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"092b1191e6d4c34ad8fc0c0e892c2b6a6358097fdf82913d1ab40599b5094a6c\": container with ID starting with 092b1191e6d4c34ad8fc0c0e892c2b6a6358097fdf82913d1ab40599b5094a6c not found: ID does not exist" containerID="092b1191e6d4c34ad8fc0c0e892c2b6a6358097fdf82913d1ab40599b5094a6c"
Nov 03 20:12:36.669461 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.669418    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"092b1191e6d4c34ad8fc0c0e892c2b6a6358097fdf82913d1ab40599b5094a6c"} err="failed to get container status \"092b1191e6d4c34ad8fc0c0e892c2b6a6358097fdf82913d1ab40599b5094a6c\": rpc error: code = NotFound desc = could not find container \"092b1191e6d4c34ad8fc0c0e892c2b6a6358097fdf82913d1ab40599b5094a6c\": container with ID starting with 092b1191e6d4c34ad8fc0c0e892c2b6a6358097fdf82913d1ab40599b5094a6c not found: ID does not exist"
Nov 03 20:12:36.704246 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.696490    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658"
Nov 03 20:12:36.735055 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.733443    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a06f29e1-b0c1-47e7-b121-783c8ce53525-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06f29e1-b0c1-47e7-b121-783c8ce53525-plugins-config-volume\") pod \"a06f29e1-b0c1-47e7-b121-783c8ce53525\" (UID: \"a06f29e1-b0c1-47e7-b121-783c8ce53525\") "
Nov 03 20:12:36.735055 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.733618    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a06f29e1-b0c1-47e7-b121-783c8ce53525-kube-api-access-xq6mg\" (UniqueName: \"kubernetes.io/projected/a06f29e1-b0c1-47e7-b121-783c8ce53525-kube-api-access-xq6mg\") pod \"a06f29e1-b0c1-47e7-b121-783c8ce53525\" (UID: \"a06f29e1-b0c1-47e7-b121-783c8ce53525\") "
Nov 03 20:12:36.735055 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.733791    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a06f29e1-b0c1-47e7-b121-783c8ce53525-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a06f29e1-b0c1-47e7-b121-783c8ce53525-provider-credentials\") pod \"a06f29e1-b0c1-47e7-b121-783c8ce53525\" (UID: \"a06f29e1-b0c1-47e7-b121-783c8ce53525\") "
Nov 03 20:12:36.735055 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.733903    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a06f29e1-b0c1-47e7-b121-783c8ce53525-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a06f29e1-b0c1-47e7-b121-783c8ce53525-routing-overlay-volume\") pod \"a06f29e1-b0c1-47e7-b121-783c8ce53525\" (UID: \"a06f29e1-b0c1-47e7-b121-783c8ce53525\") "
Nov 03 20:12:36.735055 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.734850    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdk55\" (UniqueName: \"kubernetes.io/projected/774a0a2c-2534-4a5c-9c2d-737b3aef7268-kube-api-access-rdk55\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:36.735055 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.734875    5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/774a0a2c-2534-4a5c-9c2d-737b3aef7268-routing-overlay-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:36.735055 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.734890    5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/774a0a2c-2534-4a5c-9c2d-737b3aef7268-provider-credentials\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:36.735875 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.735842    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a06f29e1-b0c1-47e7-b121-783c8ce53525-routing-overlay-volume" pod "a06f29e1-b0c1-47e7-b121-783c8ce53525" (UID: "a06f29e1-b0c1-47e7-b121-783c8ce53525"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:36.736335 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.736309    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a06f29e1-b0c1-47e7-b121-783c8ce53525-plugins-config-volume" pod "a06f29e1-b0c1-47e7-b121-783c8ce53525" (UID: "a06f29e1-b0c1-47e7-b121-783c8ce53525"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:36.740101 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.740060    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a06f29e1-b0c1-47e7-b121-783c8ce53525-provider-credentials" pod "a06f29e1-b0c1-47e7-b121-783c8ce53525" (UID: "a06f29e1-b0c1-47e7-b121-783c8ce53525"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:36.740784 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.740757    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a06f29e1-b0c1-47e7-b121-783c8ce53525-kube-api-access-xq6mg" pod "a06f29e1-b0c1-47e7-b121-783c8ce53525" (UID: "a06f29e1-b0c1-47e7-b121-783c8ce53525"). InnerVolumeSpecName "kube-api-access-xq6mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:36.836233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.836142    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq6mg\" (UniqueName: \"kubernetes.io/projected/a06f29e1-b0c1-47e7-b121-783c8ce53525-kube-api-access-xq6mg\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:36.836233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.836180    5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a06f29e1-b0c1-47e7-b121-783c8ce53525-provider-credentials\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:36.836441 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.836196    5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a06f29e1-b0c1-47e7-b121-783c8ce53525-routing-overlay-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:36.836441 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:36.836257    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a06f29e1-b0c1-47e7-b121-783c8ce53525-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:37.039229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:37.038257    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz"]
Nov 03 20:12:37.068228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:37.060245    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5dbb96dlz"]
Nov 03 20:12:37.332248 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:37.327627    5591 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 03 20:12:37.332248 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:37.327688    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds" podUID="e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 03 20:12:37.401233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:37.400541    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="774a0a2c-2534-4a5c-9c2d-737b3aef7268" path="/var/lib/kubelet/pods/774a0a2c-2534-4a5c-9c2d-737b3aef7268/volumes"
Nov 03 20:12:37.657263 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:37.657197    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658" event={"ID":"a06f29e1-b0c1-47e7-b121-783c8ce53525","Type":"ContainerDied","Data":"9d636bba134ff5bb38b15f23e1d32a9f0d237fc8dd44d22e51835eee8657d248"}
Nov 03 20:12:37.657487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:37.657285    5591 scope.go:122] "RemoveContainer" containerID="b40159a058ff67021491e6901b6a0c131ee9b6c1abc9ac0b04ebc133084c8ebf"
Nov 03 20:12:37.657487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:37.657402    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658"
Nov 03 20:12:37.719344 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:37.719256    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658"]
Nov 03 20:12:37.732810 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:37.731028    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-8495rt658"]
Nov 03 20:12:38.322556 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:38.322522    5591 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 03 20:12:38.322748 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:38.322593    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds" podUID="e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 03 20:12:38.404863 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:38.404835    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf"
Nov 03 20:12:39.321683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:39.321648    5591 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 03 20:12:39.321884 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:39.321730    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds" podUID="e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 03 20:12:39.352313 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:39.352284    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a06f29e1-b0c1-47e7-b121-783c8ce53525" path="/var/lib/kubelet/pods/a06f29e1-b0c1-47e7-b121-783c8ce53525/volumes"
Nov 03 20:12:40.321891 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:40.321855    5591 patch_prober.go:29] interesting pod/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 03 20:12:40.322414 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:40.321911    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds" podUID="e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 03 20:12:41.327616 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:41.327587    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:41.328099 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:41.328080    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:41.328817 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:41.328791    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:12:42.261303 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.261268    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"]
Nov 03 20:12:42.262172 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.262155    5591 cpu_manager.go:414] "containerMap: removing container" podUID="a06f29e1-b0c1-47e7-b121-783c8ce53525" containerName="payload-processing"
Nov 03 20:12:42.262172 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.262172    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a06f29e1-b0c1-47e7-b121-783c8ce53525" containerName="payload-processing"
Nov 03 20:12:42.262310 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.262253    5591 cpu_manager.go:414] "containerMap: removing container" podUID="774a0a2c-2534-4a5c-9c2d-737b3aef7268" containerName="payload-processing"
Nov 03 20:12:42.262310 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.262259    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="774a0a2c-2534-4a5c-9c2d-737b3aef7268" containerName="payload-processing"
Nov 03 20:12:42.262417 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.262408    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="a06f29e1-b0c1-47e7-b121-783c8ce53525" containerName="payload-processing"
Nov 03 20:12:42.262459 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.262428    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="774a0a2c-2534-4a5c-9c2d-737b3aef7268" containerName="payload-processing"
Nov 03 20:12:42.267842 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.267822    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.274296 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.274271    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0e1546-openshift-default-dockercfg-jcbdc\""
Nov 03 20:12:42.279737 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.279715    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"]
Nov 03 20:12:42.349899 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.322677    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-workload-socket\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.367419 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.363274    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-credential-socket\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.392450 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.392408    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istiod-ca-cert\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.392711 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.392694    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-token\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.392896 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.392880    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swvdr\" (UniqueName: \"kubernetes.io/projected/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-kube-api-access-swvdr\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.393123 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.393109    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-envoy\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.393243 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.393228    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-podinfo\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.393479 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.393415    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-workload-certs\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.393795 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.393768    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-data\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.494927 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.494892    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-data\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.495133 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.495021    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-workload-socket\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.495231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.495195    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-credential-socket\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.495316 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.495300    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istiod-ca-cert\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.495373 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.495347    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-token\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.495434 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.495367    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-data\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.495493 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.495462    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-workload-socket\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.496224 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.496165    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swvdr\" (UniqueName: \"kubernetes.io/projected/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-kube-api-access-swvdr\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.496354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.496247    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-credential-socket\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.496354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.496286    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-envoy\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.496354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.496316    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-podinfo\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.496354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.496345    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-workload-certs\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.496761 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.496730    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istiod-ca-cert\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.496882 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.496768    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-workload-certs\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.499429 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.499403    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-envoy\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.499572 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.499493    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-podinfo\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.509527 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.509494    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-token\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.512668 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.512600    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swvdr\" (UniqueName: \"kubernetes.io/projected/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-kube-api-access-swvdr\") pod \"e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") " pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.586852 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.586816    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:42.911409 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.911379    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"]
Nov 03 20:12:42.911779 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:12:42.911752    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a55d034_3d5e_41f1_ae14_b741d8b0c2a5.slice/crio-092781921a56ad0b953512dd5fa8ec0b478d505e64bc57cf62f7cdae07a66f98 WatchSource:0}: Error finding container 092781921a56ad0b953512dd5fa8ec0b478d505e64bc57cf62f7cdae07a66f98: Status 404 returned error can't find the container with id 092781921a56ad0b953512dd5fa8ec0b478d505e64bc57cf62f7cdae07a66f98
Nov 03 20:12:42.915293 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.915197    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:42.915415 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.915342    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:42.915461 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:42.915414    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:12:43.742923 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:43.742881    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l" event={"ID":"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5","Type":"ContainerStarted","Data":"3614abd781be46c274bc81b850b2a40344ae5f0eba3a5bfb56dd0d3d2d911d46"}
Nov 03 20:12:43.743456 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:43.742947    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l" event={"ID":"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5","Type":"ContainerStarted","Data":"092781921a56ad0b953512dd5fa8ec0b478d505e64bc57cf62f7cdae07a66f98"}
Nov 03 20:12:43.771584 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:43.771500    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l" podStartSLOduration=1.7714848 podStartE2EDuration="1.7714848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:43.767033531 +0000 UTC m=+2112.539326573" watchObservedRunningTime="2026-11-03 20:12:43.7714848 +0000 UTC m=+2112.543777817"
Nov 03 20:12:44.595582 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:44.594599    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:44.606310 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:44.606193    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 03 20:12:44.606536 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:44.606347    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l" podUID="4a55d034-3d5e-41f1-ae14-b741d8b0c2a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 03 20:12:44.963377 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:44.963273    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln"]
Nov 03 20:12:44.970358 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:44.970324    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln"
Nov 03 20:12:45.007642 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:45.007589    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-4c2a28-serving-cert\""
Nov 03 20:12:45.013739 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:45.013704    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln"]
Nov 03 20:12:45.086232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:45.069324    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86168259-d196-495e-9435-33d4f0e625d9-metrics-certs\") pod \"maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln\" (UID: \"86168259-d196-495e-9435-33d4f0e625d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln"
Nov 03 20:12:45.086232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:45.069501    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86168259-d196-495e-9435-33d4f0e625d9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln\" (UID: \"86168259-d196-495e-9435-33d4f0e625d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln"
Nov 03 20:12:45.086232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:45.069780    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztkxt\" (UniqueName: \"kubernetes.io/projected/86168259-d196-495e-9435-33d4f0e625d9-kube-api-access-ztkxt\") pod \"maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln\" (UID: \"86168259-d196-495e-9435-33d4f0e625d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln"
Nov 03 20:12:45.086232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:45.070006    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86168259-d196-495e-9435-33d4f0e625d9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln\" (UID: \"86168259-d196-495e-9435-33d4f0e625d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln"
Nov 03 20:12:45.176479 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:45.176427    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztkxt\" (UniqueName: \"kubernetes.io/projected/86168259-d196-495e-9435-33d4f0e625d9-kube-api-access-ztkxt\") pod \"maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln\" (UID: \"86168259-d196-495e-9435-33d4f0e625d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln"
Nov 03 20:12:45.180287 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:45.180252    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86168259-d196-495e-9435-33d4f0e625d9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln\" (UID: \"86168259-d196-495e-9435-33d4f0e625d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln"
Nov 03 20:12:45.180821 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:45.180783    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86168259-d196-495e-9435-33d4f0e625d9-metrics-certs\") pod \"maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln\" (UID: \"86168259-d196-495e-9435-33d4f0e625d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln"
Nov 03 20:12:45.181149 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:45.181129    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86168259-d196-495e-9435-33d4f0e625d9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln\" (UID: \"86168259-d196-495e-9435-33d4f0e625d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln"
Nov 03 20:12:45.181784 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:12:45.181761    5591 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-serving-cert: secret "maas-api-e2e-extmod-b-4c2a28-serving-cert" not found
Nov 03 20:12:45.189924 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:45.182865    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86168259-d196-495e-9435-33d4f0e625d9-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln\" (UID: \"86168259-d196-495e-9435-33d4f0e625d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln"
Nov 03 20:12:45.190290 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:12:45.190266    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86168259-d196-495e-9435-33d4f0e625d9-maas-api-tls podName:86168259-d196-495e-9435-33d4f0e625d9 nodeName:}" failed. No retries permitted until 2026-11-03 20:12:45.681966308 +0000 UTC m=+2114.454259320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/86168259-d196-495e-9435-33d4f0e625d9-maas-api-tls") pod "maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln" (UID: "86168259-d196-495e-9435-33d4f0e625d9") : secret "maas-api-e2e-extmod-b-4c2a28-serving-cert" not found
Nov 03 20:12:45.244559 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:45.244440    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86168259-d196-495e-9435-33d4f0e625d9-metrics-certs\") pod \"maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln\" (UID: \"86168259-d196-495e-9435-33d4f0e625d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln"
Nov 03 20:12:45.244753 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:45.244598    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztkxt\" (UniqueName: \"kubernetes.io/projected/86168259-d196-495e-9435-33d4f0e625d9-kube-api-access-ztkxt\") pod \"maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln\" (UID: \"86168259-d196-495e-9435-33d4f0e625d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln"
Nov 03 20:12:45.595281 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:45.589875    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 03 20:12:45.595281 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:45.589942    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l" podUID="4a55d034-3d5e-41f1-ae14-b741d8b0c2a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 03 20:12:45.715751 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:45.715688    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86168259-d196-495e-9435-33d4f0e625d9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln\" (UID: \"86168259-d196-495e-9435-33d4f0e625d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln"
Nov 03 20:12:45.759229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:45.757944    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86168259-d196-495e-9435-33d4f0e625d9-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln\" (UID: \"86168259-d196-495e-9435-33d4f0e625d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln"
Nov 03 20:12:45.942290 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:45.938512    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln"
Nov 03 20:12:46.314694 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:46.314662    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln"]
Nov 03 20:12:46.320152 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:12:46.319688    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86168259_d196_495e_9435_33d4f0e625d9.slice/crio-6da58e61f9a4a1d5811ae5ee663ca359b4624f52ad916ce31cf17dd4647fb420 WatchSource:0}: Error finding container 6da58e61f9a4a1d5811ae5ee663ca359b4624f52ad916ce31cf17dd4647fb420: Status 404 returned error can't find the container with id 6da58e61f9a4a1d5811ae5ee663ca359b4624f52ad916ce31cf17dd4647fb420
Nov 03 20:12:46.589324 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:46.587314    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 03 20:12:46.589324 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:46.587374    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l" podUID="4a55d034-3d5e-41f1-ae14-b741d8b0c2a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 03 20:12:46.962990 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:46.959872    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln" event={"ID":"86168259-d196-495e-9435-33d4f0e625d9","Type":"ContainerStarted","Data":"6da58e61f9a4a1d5811ae5ee663ca359b4624f52ad916ce31cf17dd4647fb420"}
Nov 03 20:12:47.601962 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:47.601905    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 03 20:12:47.602464 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:47.601979    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l" podUID="4a55d034-3d5e-41f1-ae14-b741d8b0c2a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 03 20:12:48.060430 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:48.055886    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln" event={"ID":"86168259-d196-495e-9435-33d4f0e625d9","Type":"ContainerStarted","Data":"0d95a0f4c09a0e24cdfc1de8d3a706147ebab212c830ad14ac999d77db8fb133"}
Nov 03 20:12:48.060430 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:48.059495    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln"
Nov 03 20:12:48.100229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:48.099147    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln" podStartSLOduration=3.465366964 podStartE2EDuration="4.099125392s" totalImagesPullingTime="633.758428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:48.085733011 +0000 UTC m=+2116.858026030" watchObservedRunningTime="2026-11-03 20:12:48.099125392 +0000 UTC m=+2116.871418415"
Nov 03 20:12:48.604910 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:48.594936    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 03 20:12:48.604910 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:48.595006    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l" podUID="4a55d034-3d5e-41f1-ae14-b741d8b0c2a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 03 20:12:49.601160 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:49.601082    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 03 20:12:49.601603 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:49.601569    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l" podUID="4a55d034-3d5e-41f1-ae14-b741d8b0c2a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 03 20:12:50.593769 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:50.593739    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:50.594241 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:50.594038    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:50.594939 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:50.594915    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:12:53.896073 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:53.896035    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf"]
Nov 03 20:12:54.094502 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:54.094450    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln"
Nov 03 20:12:58.731041 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:12:58.730996    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:12:58.897144 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:58.897094    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf" podUID="2c30c58b-4263-430d-acd1-0b93599407c6" containerName="payload-processing" containerID="cri-o://1f07e5b56d46870fd44192d731c59019220bd7a92ed2ca5f8826c6ffe270e890" gracePeriod=25
Nov 03 20:12:59.195284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:59.195254    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf"
Nov 03 20:12:59.292945 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:59.292859    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c30c58b-4263-430d-acd1-0b93599407c6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c30c58b-4263-430d-acd1-0b93599407c6-provider-credentials\") pod \"2c30c58b-4263-430d-acd1-0b93599407c6\" (UID: \"2c30c58b-4263-430d-acd1-0b93599407c6\") "
Nov 03 20:12:59.293713 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:59.293687    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c30c58b-4263-430d-acd1-0b93599407c6-kube-api-access-fjlqv\" (UniqueName: \"kubernetes.io/projected/2c30c58b-4263-430d-acd1-0b93599407c6-kube-api-access-fjlqv\") pod \"2c30c58b-4263-430d-acd1-0b93599407c6\" (UID: \"2c30c58b-4263-430d-acd1-0b93599407c6\") "
Nov 03 20:12:59.293870 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:59.293754    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c30c58b-4263-430d-acd1-0b93599407c6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c30c58b-4263-430d-acd1-0b93599407c6-routing-overlay-volume\") pod \"2c30c58b-4263-430d-acd1-0b93599407c6\" (UID: \"2c30c58b-4263-430d-acd1-0b93599407c6\") "
Nov 03 20:12:59.293870 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:59.293800    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c30c58b-4263-430d-acd1-0b93599407c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c30c58b-4263-430d-acd1-0b93599407c6-plugins-config-volume\") pod \"2c30c58b-4263-430d-acd1-0b93599407c6\" (UID: \"2c30c58b-4263-430d-acd1-0b93599407c6\") "
Nov 03 20:12:59.294617 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:59.294586    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c30c58b-4263-430d-acd1-0b93599407c6-routing-overlay-volume" pod "2c30c58b-4263-430d-acd1-0b93599407c6" (UID: "2c30c58b-4263-430d-acd1-0b93599407c6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:59.294811 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:59.294722    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c30c58b-4263-430d-acd1-0b93599407c6-plugins-config-volume" pod "2c30c58b-4263-430d-acd1-0b93599407c6" (UID: "2c30c58b-4263-430d-acd1-0b93599407c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:12:59.310257 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:59.307251    5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c30c58b-4263-430d-acd1-0b93599407c6-routing-overlay-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:59.310257 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:59.307291    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c30c58b-4263-430d-acd1-0b93599407c6-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:59.336236 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:59.331927    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c30c58b-4263-430d-acd1-0b93599407c6-provider-credentials" pod "2c30c58b-4263-430d-acd1-0b93599407c6" (UID: "2c30c58b-4263-430d-acd1-0b93599407c6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:59.350278 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:59.343659    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c30c58b-4263-430d-acd1-0b93599407c6-kube-api-access-fjlqv" pod "2c30c58b-4263-430d-acd1-0b93599407c6" (UID: "2c30c58b-4263-430d-acd1-0b93599407c6"). InnerVolumeSpecName "kube-api-access-fjlqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:12:59.409469 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:59.409440    5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c30c58b-4263-430d-acd1-0b93599407c6-provider-credentials\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:59.409469 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:59.409468    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjlqv\" (UniqueName: \"kubernetes.io/projected/2c30c58b-4263-430d-acd1-0b93599407c6-kube-api-access-fjlqv\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:12:59.481159 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:59.481124    5591 generic.go:356] "Generic (PLEG): container finished" podID="2c30c58b-4263-430d-acd1-0b93599407c6" containerID="1f07e5b56d46870fd44192d731c59019220bd7a92ed2ca5f8826c6ffe270e890" exitCode=0
Nov 03 20:12:59.481370 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:59.481167    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf"
Nov 03 20:12:59.481370 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:59.481251    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf" event={"ID":"2c30c58b-4263-430d-acd1-0b93599407c6","Type":"ContainerDied","Data":"1f07e5b56d46870fd44192d731c59019220bd7a92ed2ca5f8826c6ffe270e890"}
Nov 03 20:12:59.481370 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:59.481303    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf" event={"ID":"2c30c58b-4263-430d-acd1-0b93599407c6","Type":"ContainerDied","Data":"66244fd2dc3b33d9828a5625712944362f5b46e01974566baf47edbbf3d3cdd1"}
Nov 03 20:12:59.481370 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:59.481325    5591 scope.go:122] "RemoveContainer" containerID="1f07e5b56d46870fd44192d731c59019220bd7a92ed2ca5f8826c6ffe270e890"
Nov 03 20:12:59.492960 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:59.492937    5591 scope.go:122] "RemoveContainer" containerID="1f07e5b56d46870fd44192d731c59019220bd7a92ed2ca5f8826c6ffe270e890"
Nov 03 20:12:59.493257 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:12:59.493233    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f07e5b56d46870fd44192d731c59019220bd7a92ed2ca5f8826c6ffe270e890\": container with ID starting with 1f07e5b56d46870fd44192d731c59019220bd7a92ed2ca5f8826c6ffe270e890 not found: ID does not exist" containerID="1f07e5b56d46870fd44192d731c59019220bd7a92ed2ca5f8826c6ffe270e890"
Nov 03 20:12:59.493355 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:59.493265    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f07e5b56d46870fd44192d731c59019220bd7a92ed2ca5f8826c6ffe270e890"} err="failed to get container status \"1f07e5b56d46870fd44192d731c59019220bd7a92ed2ca5f8826c6ffe270e890\": rpc error: code = NotFound desc = could not find container \"1f07e5b56d46870fd44192d731c59019220bd7a92ed2ca5f8826c6ffe270e890\": container with ID starting with 1f07e5b56d46870fd44192d731c59019220bd7a92ed2ca5f8826c6ffe270e890 not found: ID does not exist"
Nov 03 20:12:59.505400 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:59.505373    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf"]
Nov 03 20:12:59.511324 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:12:59.511293    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b503a8/payload-processing-external-model-e2e-extmod-b-b503a8-6d6cn2kgf"]
Nov 03 20:13:00.268588 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:00.268557    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv"]
Nov 03 20:13:00.269711 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:00.269689    5591 cpu_manager.go:414] "containerMap: removing container" podUID="2c30c58b-4263-430d-acd1-0b93599407c6" containerName="payload-processing"
Nov 03 20:13:00.269777 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:00.269715    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c30c58b-4263-430d-acd1-0b93599407c6" containerName="payload-processing"
Nov 03 20:13:00.269959 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:00.269946    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c30c58b-4263-430d-acd1-0b93599407c6" containerName="payload-processing"
Nov 03 20:13:00.307229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:00.301120    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv"
Nov 03 20:13:00.405663 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:00.398633    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"payload-processing-external-model-e2e--a1bcdb2a-dockercfg-5sh7w\""
Nov 03 20:13:00.405663 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:00.398692    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"agc-cred-9d0b4d393cba\""
Nov 03 20:13:00.405663 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:00.398803    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"kube-root-ca.crt\""
Nov 03 20:13:00.405663 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:00.398863    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"routing-overlay\""
Nov 03 20:13:00.405663 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:00.398905    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"openshift-service-ca.crt\""
Nov 03 20:13:00.405663 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:00.398962    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-46a6d7\"/\"payload-processing-external-model-plugins-e2e-extmod-b-46a6d7\""
Nov 03 20:13:00.568338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:00.560345    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv\" (UID: \"8609c94b-9a4c-4335-8d07-4bfbf000b0e6\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv"
Nov 03 20:13:00.568338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:00.560469    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzw5x\" (UniqueName: \"kubernetes.io/projected/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-kube-api-access-fzw5x\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv\" (UID: \"8609c94b-9a4c-4335-8d07-4bfbf000b0e6\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv"
Nov 03 20:13:00.568338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:00.560585    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv\" (UID: \"8609c94b-9a4c-4335-8d07-4bfbf000b0e6\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv"
Nov 03 20:13:00.568338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:00.561134    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv\" (UID: \"8609c94b-9a4c-4335-8d07-4bfbf000b0e6\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv"
Nov 03 20:13:00.588249 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:00.586338    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv"]
Nov 03 20:13:00.671331 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:00.671276    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv\" (UID: \"8609c94b-9a4c-4335-8d07-4bfbf000b0e6\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv"
Nov 03 20:13:00.673261 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:00.671684    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv\" (UID: \"8609c94b-9a4c-4335-8d07-4bfbf000b0e6\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv"
Nov 03 20:13:00.673261 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:00.671790    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzw5x\" (UniqueName: \"kubernetes.io/projected/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-kube-api-access-fzw5x\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv\" (UID: \"8609c94b-9a4c-4335-8d07-4bfbf000b0e6\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv"
Nov 03 20:13:00.673261 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:00.672005    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv\" (UID: \"8609c94b-9a4c-4335-8d07-4bfbf000b0e6\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv"
Nov 03 20:13:00.684954 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:00.684922    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv\" (UID: \"8609c94b-9a4c-4335-8d07-4bfbf000b0e6\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv"
Nov 03 20:13:00.685242 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:00.685195    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv\" (UID: \"8609c94b-9a4c-4335-8d07-4bfbf000b0e6\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv"
Nov 03 20:13:00.690414 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:00.690381    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv\" (UID: \"8609c94b-9a4c-4335-8d07-4bfbf000b0e6\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv"
Nov 03 20:13:00.700331 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:00.700167    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzw5x\" (UniqueName: \"kubernetes.io/projected/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-kube-api-access-fzw5x\") pod \"payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv\" (UID: \"8609c94b-9a4c-4335-8d07-4bfbf000b0e6\") " pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv"
Nov 03 20:13:00.742874 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:00.742841    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv"
Nov 03 20:13:01.039484 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:13:01.039445    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8609c94b_9a4c_4335_8d07_4bfbf000b0e6.slice/crio-b794aef8bdd629dcdb11874f5aa7ae2c29fac9644abb26468a1d25d9c41f35f8 WatchSource:0}: Error finding container b794aef8bdd629dcdb11874f5aa7ae2c29fac9644abb26468a1d25d9c41f35f8: Status 404 returned error can't find the container with id b794aef8bdd629dcdb11874f5aa7ae2c29fac9644abb26468a1d25d9c41f35f8
Nov 03 20:13:01.040096 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:01.039957    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv"]
Nov 03 20:13:01.421447 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:01.419344    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c30c58b-4263-430d-acd1-0b93599407c6" path="/var/lib/kubelet/pods/2c30c58b-4263-430d-acd1-0b93599407c6/volumes"
Nov 03 20:13:01.686966 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:01.686878    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv" event={"ID":"8609c94b-9a4c-4335-8d07-4bfbf000b0e6","Type":"ContainerStarted","Data":"b967ec3bcc13e02c7d960026dce586ab81858e0b206f7925e3c4f6d3d71d71e0"}
Nov 03 20:13:01.686966 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:01.686943    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv" event={"ID":"8609c94b-9a4c-4335-8d07-4bfbf000b0e6","Type":"ContainerStarted","Data":"b794aef8bdd629dcdb11874f5aa7ae2c29fac9644abb26468a1d25d9c41f35f8"}
Nov 03 20:13:01.687170 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:01.686976    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv"
Nov 03 20:13:01.732057 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:01.731993    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv" podStartSLOduration=1.7319792170000001 podStartE2EDuration="1.731979217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:01.730551402 +0000 UTC m=+2130.502844419" watchObservedRunningTime="2026-11-03 20:13:01.731979217 +0000 UTC m=+2130.504272234"
Nov 03 20:13:12.698886 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:12.698850    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv"
Nov 03 20:13:17.329933 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:17.329888    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:13:28.728565 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:13:28.728525    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:13:39.263963 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.263924    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"]
Nov 03 20:13:39.270828 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.270801    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.277347 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.277319    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0e4ce7-openshift-default-dockercfg-nppzh\""
Nov 03 20:13:39.283706 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.283673    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"]
Nov 03 20:13:39.362104 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.362069    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-workload-certs\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.362342 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.362234    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-workload-socket\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.362342 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.362303    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-token\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.362475 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.362427    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-data\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.362475 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.362464    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-envoy\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.362599 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.362509    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rx64\" (UniqueName: \"kubernetes.io/projected/b7189d7d-2381-4085-94b8-5b6569b7da99-kube-api-access-9rx64\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.362672 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.362643    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-podinfo\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.362730 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.362701    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-credential-socket\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.362836 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.362812    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7189d7d-2381-4085-94b8-5b6569b7da99-istiod-ca-cert\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.464806 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.464766    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-workload-socket\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.464806 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.464806    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-token\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.465107 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.464904    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-data\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.465107 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.464933    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-envoy\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.465107 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.464964    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rx64\" (UniqueName: \"kubernetes.io/projected/b7189d7d-2381-4085-94b8-5b6569b7da99-kube-api-access-9rx64\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.465107 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.465042    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-podinfo\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.465107 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.465072    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-credential-socket\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.465444 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.465177    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7189d7d-2381-4085-94b8-5b6569b7da99-istiod-ca-cert\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.465444 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.465414    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-workload-certs\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.466519 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.465802    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-workload-socket\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.466944 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.466918    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-data\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.467088 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.466943    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-workload-certs\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.467088 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.467058    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-credential-socket\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.467752 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.467728    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7189d7d-2381-4085-94b8-5b6569b7da99-istiod-ca-cert\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.469415 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.469383    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-envoy\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.469415 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.469399    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-podinfo\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.480649 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.480614    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-token\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.481002 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.480978    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rx64\" (UniqueName: \"kubernetes.io/projected/b7189d7d-2381-4085-94b8-5b6569b7da99-kube-api-access-9rx64\") pod \"e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") " pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:39.597064 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:39.596975    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:40.221227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:40.220548    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"]
Nov 03 20:13:40.284228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:40.282876    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:13:40.284228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:40.282983    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:13:40.284228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:40.283025    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:13:40.802709 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:40.802671    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh" event={"ID":"b7189d7d-2381-4085-94b8-5b6569b7da99","Type":"ContainerStarted","Data":"1b323464aeb036e8bf2ea02ff3ea6cfd32b7b942acd495215959e915c4e0be9d"}
Nov 03 20:13:40.802919 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:40.802725    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh" event={"ID":"b7189d7d-2381-4085-94b8-5b6569b7da99","Type":"ContainerStarted","Data":"56e528913aeb18abc437cc93f5296433d5c5ea6ec89c35ad85dba6339be90400"}
Nov 03 20:13:40.829536 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:40.829476    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh" podStartSLOduration=1.829460324 podStartE2EDuration="1.829460324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:40.827823459 +0000 UTC m=+2169.600116469" watchObservedRunningTime="2026-11-03 20:13:40.829460324 +0000 UTC m=+2169.601753343"
Nov 03 20:13:41.606039 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:41.605793    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:41.607397 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:41.607360    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body=
Nov 03 20:13:41.607544 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:41.607419    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh" podUID="b7189d7d-2381-4085-94b8-5b6569b7da99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused"
Nov 03 20:13:42.604370 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:42.604333    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body=
Nov 03 20:13:42.605129 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:42.605094    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh" podUID="b7189d7d-2381-4085-94b8-5b6569b7da99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused"
Nov 03 20:13:43.598822 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:43.598789    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body=
Nov 03 20:13:43.599625 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:43.599592    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh" podUID="b7189d7d-2381-4085-94b8-5b6569b7da99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused"
Nov 03 20:13:43.728151 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:43.728116    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv"]
Nov 03 20:13:44.604868 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:44.600575    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body=
Nov 03 20:13:44.604868 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:44.600639    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh" podUID="b7189d7d-2381-4085-94b8-5b6569b7da99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused"
Nov 03 20:13:45.599460 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:45.599418    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body=
Nov 03 20:13:45.599858 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:45.599823    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh" podUID="b7189d7d-2381-4085-94b8-5b6569b7da99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused"
Nov 03 20:13:46.601864 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:46.601829    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body=
Nov 03 20:13:46.602530 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:46.602496    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh" podUID="b7189d7d-2381-4085-94b8-5b6569b7da99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused"
Nov 03 20:13:47.604312 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:47.604280    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:47.604790 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:47.604768    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:47.605711 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:47.605687    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:13:48.729072 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:48.729023    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv" podUID="8609c94b-9a4c-4335-8d07-4bfbf000b0e6" containerName="payload-processing" containerID="cri-o://b967ec3bcc13e02c7d960026dce586ab81858e0b206f7925e3c4f6d3d71d71e0" gracePeriod=25
Nov 03 20:13:48.920230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:48.916258    5591 generic.go:356] "Generic (PLEG): container finished" podID="8609c94b-9a4c-4335-8d07-4bfbf000b0e6" containerID="b967ec3bcc13e02c7d960026dce586ab81858e0b206f7925e3c4f6d3d71d71e0" exitCode=0
Nov 03 20:13:48.920230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:48.917755    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv" event={"ID":"8609c94b-9a4c-4335-8d07-4bfbf000b0e6","Type":"ContainerDied","Data":"b967ec3bcc13e02c7d960026dce586ab81858e0b206f7925e3c4f6d3d71d71e0"}
Nov 03 20:13:49.115564 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:49.115533    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv"
Nov 03 20:13:49.214610 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:49.214574    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-routing-overlay-volume\") pod \"8609c94b-9a4c-4335-8d07-4bfbf000b0e6\" (UID: \"8609c94b-9a4c-4335-8d07-4bfbf000b0e6\") "
Nov 03 20:13:49.214802 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:49.214779    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-provider-credentials\") pod \"8609c94b-9a4c-4335-8d07-4bfbf000b0e6\" (UID: \"8609c94b-9a4c-4335-8d07-4bfbf000b0e6\") "
Nov 03 20:13:49.214877 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:49.214842    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-plugins-config-volume\") pod \"8609c94b-9a4c-4335-8d07-4bfbf000b0e6\" (UID: \"8609c94b-9a4c-4335-8d07-4bfbf000b0e6\") "
Nov 03 20:13:49.214937 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:49.214913    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-kube-api-access-fzw5x\" (UniqueName: \"kubernetes.io/projected/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-kube-api-access-fzw5x\") pod \"8609c94b-9a4c-4335-8d07-4bfbf000b0e6\" (UID: \"8609c94b-9a4c-4335-8d07-4bfbf000b0e6\") "
Nov 03 20:13:49.215031 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:49.214999    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-routing-overlay-volume" pod "8609c94b-9a4c-4335-8d07-4bfbf000b0e6" (UID: "8609c94b-9a4c-4335-8d07-4bfbf000b0e6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:49.215349 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:49.215304    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-plugins-config-volume" pod "8609c94b-9a4c-4335-8d07-4bfbf000b0e6" (UID: "8609c94b-9a4c-4335-8d07-4bfbf000b0e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:49.215750 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:49.215725    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:49.215818 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:49.215756    5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-routing-overlay-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:49.217787 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:49.217763    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-provider-credentials" pod "8609c94b-9a4c-4335-8d07-4bfbf000b0e6" (UID: "8609c94b-9a4c-4335-8d07-4bfbf000b0e6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:49.217886 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:49.217804    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-kube-api-access-fzw5x" pod "8609c94b-9a4c-4335-8d07-4bfbf000b0e6" (UID: "8609c94b-9a4c-4335-8d07-4bfbf000b0e6"). InnerVolumeSpecName "kube-api-access-fzw5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:49.317367 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:49.317323    5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-provider-credentials\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:49.317367 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:49.317363    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzw5x\" (UniqueName: \"kubernetes.io/projected/8609c94b-9a4c-4335-8d07-4bfbf000b0e6-kube-api-access-fzw5x\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:49.928155 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:49.928124    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv"
Nov 03 20:13:49.928645 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:49.928150    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv" event={"ID":"8609c94b-9a4c-4335-8d07-4bfbf000b0e6","Type":"ContainerDied","Data":"b794aef8bdd629dcdb11874f5aa7ae2c29fac9644abb26468a1d25d9c41f35f8"}
Nov 03 20:13:49.928645 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:49.928254    5591 scope.go:122] "RemoveContainer" containerID="b967ec3bcc13e02c7d960026dce586ab81858e0b206f7925e3c4f6d3d71d71e0"
Nov 03 20:13:49.958731 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:49.958687    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv"]
Nov 03 20:13:49.966013 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:49.965970    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-46a6d7/payload-processing-external-model-e2e-extmod-b-46a6d7-6799xrplv"]
Nov 03 20:13:51.279952 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:51.279917    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg"]
Nov 03 20:13:51.354112 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:51.354079    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8609c94b-9a4c-4335-8d07-4bfbf000b0e6" path="/var/lib/kubelet/pods/8609c94b-9a4c-4335-8d07-4bfbf000b0e6/volumes"
Nov 03 20:13:56.280959 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:56.280910    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg" podUID="330b5625-e53f-4c73-93cb-98889bc2f0b8" containerName="payload-processing" containerID="cri-o://9896e99c9600cfd9376ed843627e7aad02cc341a3d12a4764abe126eed2e7162" gracePeriod=25
Nov 03 20:13:56.972780 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:56.972743    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg"
Nov 03 20:13:57.014676 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:57.014642    5591 generic.go:356] "Generic (PLEG): container finished" podID="330b5625-e53f-4c73-93cb-98889bc2f0b8" containerID="9896e99c9600cfd9376ed843627e7aad02cc341a3d12a4764abe126eed2e7162" exitCode=0
Nov 03 20:13:57.015255 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:57.015227    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg" event={"ID":"330b5625-e53f-4c73-93cb-98889bc2f0b8","Type":"ContainerDied","Data":"9896e99c9600cfd9376ed843627e7aad02cc341a3d12a4764abe126eed2e7162"}
Nov 03 20:13:57.015451 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:57.015436    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg" event={"ID":"330b5625-e53f-4c73-93cb-98889bc2f0b8","Type":"ContainerDied","Data":"3292c1715b74bfd7b52729d0f5bab59a94349368b982111f570c06c7337904b1"}
Nov 03 20:13:57.015567 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:57.015555    5591 scope.go:122] "RemoveContainer" containerID="9896e99c9600cfd9376ed843627e7aad02cc341a3d12a4764abe126eed2e7162"
Nov 03 20:13:57.015811 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:57.015795    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg"
Nov 03 20:13:57.161231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:57.153660    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/330b5625-e53f-4c73-93cb-98889bc2f0b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/330b5625-e53f-4c73-93cb-98889bc2f0b8-plugins-config-volume\") pod \"330b5625-e53f-4c73-93cb-98889bc2f0b8\" (UID: \"330b5625-e53f-4c73-93cb-98889bc2f0b8\") "
Nov 03 20:13:57.201068 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:57.177896    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/330b5625-e53f-4c73-93cb-98889bc2f0b8-plugins-config-volume" pod "330b5625-e53f-4c73-93cb-98889bc2f0b8" (UID: "330b5625-e53f-4c73-93cb-98889bc2f0b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:57.224829 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:57.224658    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/330b5625-e53f-4c73-93cb-98889bc2f0b8-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:57.415707 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:57.413408    5591 scope.go:122] "RemoveContainer" containerID="9896e99c9600cfd9376ed843627e7aad02cc341a3d12a4764abe126eed2e7162"
Nov 03 20:13:57.415707 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:57.414280    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/330b5625-e53f-4c73-93cb-98889bc2f0b8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/330b5625-e53f-4c73-93cb-98889bc2f0b8-provider-credentials\") pod \"330b5625-e53f-4c73-93cb-98889bc2f0b8\" (UID: \"330b5625-e53f-4c73-93cb-98889bc2f0b8\") "
Nov 03 20:13:57.415707 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:57.414411    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/330b5625-e53f-4c73-93cb-98889bc2f0b8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/330b5625-e53f-4c73-93cb-98889bc2f0b8-routing-overlay-volume\") pod \"330b5625-e53f-4c73-93cb-98889bc2f0b8\" (UID: \"330b5625-e53f-4c73-93cb-98889bc2f0b8\") "
Nov 03 20:13:57.415707 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:57.414496    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/330b5625-e53f-4c73-93cb-98889bc2f0b8-kube-api-access-gst8n\" (UniqueName: \"kubernetes.io/projected/330b5625-e53f-4c73-93cb-98889bc2f0b8-kube-api-access-gst8n\") pod \"330b5625-e53f-4c73-93cb-98889bc2f0b8\" (UID: \"330b5625-e53f-4c73-93cb-98889bc2f0b8\") "
Nov 03 20:13:57.454099 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:13:57.454051    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9896e99c9600cfd9376ed843627e7aad02cc341a3d12a4764abe126eed2e7162\": container with ID starting with 9896e99c9600cfd9376ed843627e7aad02cc341a3d12a4764abe126eed2e7162 not found: ID does not exist" containerID="9896e99c9600cfd9376ed843627e7aad02cc341a3d12a4764abe126eed2e7162"
Nov 03 20:13:57.479284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:57.470939    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/330b5625-e53f-4c73-93cb-98889bc2f0b8-routing-overlay-volume" pod "330b5625-e53f-4c73-93cb-98889bc2f0b8" (UID: "330b5625-e53f-4c73-93cb-98889bc2f0b8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:13:57.479284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:57.456350    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9896e99c9600cfd9376ed843627e7aad02cc341a3d12a4764abe126eed2e7162"} err="failed to get container status \"9896e99c9600cfd9376ed843627e7aad02cc341a3d12a4764abe126eed2e7162\": rpc error: code = NotFound desc = could not find container \"9896e99c9600cfd9376ed843627e7aad02cc341a3d12a4764abe126eed2e7162\": container with ID starting with 9896e99c9600cfd9376ed843627e7aad02cc341a3d12a4764abe126eed2e7162 not found: ID does not exist"
Nov 03 20:13:57.524310 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:57.521338    5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/330b5625-e53f-4c73-93cb-98889bc2f0b8-routing-overlay-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:57.546116 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:57.541287    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330b5625-e53f-4c73-93cb-98889bc2f0b8-kube-api-access-gst8n" pod "330b5625-e53f-4c73-93cb-98889bc2f0b8" (UID: "330b5625-e53f-4c73-93cb-98889bc2f0b8"). InnerVolumeSpecName "kube-api-access-gst8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:57.549104 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:57.547850    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330b5625-e53f-4c73-93cb-98889bc2f0b8-provider-credentials" pod "330b5625-e53f-4c73-93cb-98889bc2f0b8" (UID: "330b5625-e53f-4c73-93cb-98889bc2f0b8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:13:57.623543 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:57.623502    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gst8n\" (UniqueName: \"kubernetes.io/projected/330b5625-e53f-4c73-93cb-98889bc2f0b8-kube-api-access-gst8n\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:57.623543 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:57.623546    5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/330b5625-e53f-4c73-93cb-98889bc2f0b8-provider-credentials\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:13:57.662427 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:57.662337    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg"]
Nov 03 20:13:57.676070 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:57.676016    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0e4ce7/payload-processing-external-model-e2e-extmod-b-0e4ce7-5b47bh9hg"]
Nov 03 20:13:58.737771 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:13:58.737723    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:13:59.352164 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:13:59.352128    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="330b5625-e53f-4c73-93cb-98889bc2f0b8" path="/var/lib/kubelet/pods/330b5625-e53f-4c73-93cb-98889bc2f0b8/volumes"
Nov 03 20:14:21.455315 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:21.455256    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"]
Nov 03 20:14:21.455982 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:21.455572    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n" podUID="2b1c87e7-ecf0-438f-a3c4-93da53813516" containerName="istio-proxy" containerID="cri-o://968872f53ed39a1ad4244ad035af2045618c1bf05eedd7b4b81b430ffb55b8cf" gracePeriod=30
Nov 03 20:14:21.985293 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:21.980448    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"]
Nov 03 20:14:21.985293 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:21.980751    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7" podUID="3c70d023-1700-42d0-964f-5573383d1355" containerName="istio-proxy" containerID="cri-o://456c1f059ed2fdcef06f23f5cf106c7ca251c32337d4d7794f0d6c13e96268b9" gracePeriod=30
Nov 03 20:14:26.835401 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:26.835370    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:14:26.942842 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:26.942801    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-podinfo\") pod \"2b1c87e7-ecf0-438f-a3c4-93da53813516\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") "
Nov 03 20:14:26.943040 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:26.942852    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-data\") pod \"2b1c87e7-ecf0-438f-a3c4-93da53813516\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") "
Nov 03 20:14:26.943126 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:26.943036    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-envoy\") pod \"2b1c87e7-ecf0-438f-a3c4-93da53813516\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") "
Nov 03 20:14:26.943183 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:26.943122    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-credential-socket\") pod \"2b1c87e7-ecf0-438f-a3c4-93da53813516\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") "
Nov 03 20:14:26.943302 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:26.943272    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-token\" (UniqueName: \"kubernetes.io/projected/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-token\") pod \"2b1c87e7-ecf0-438f-a3c4-93da53813516\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") "
Nov 03 20:14:26.943302 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:26.943274    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-data" pod "2b1c87e7-ecf0-438f-a3c4-93da53813516" (UID: "2b1c87e7-ecf0-438f-a3c4-93da53813516"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:14:26.943437 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:26.943321    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b1c87e7-ecf0-438f-a3c4-93da53813516-kube-api-access-dpccm\" (UniqueName: \"kubernetes.io/projected/2b1c87e7-ecf0-438f-a3c4-93da53813516-kube-api-access-dpccm\") pod \"2b1c87e7-ecf0-438f-a3c4-93da53813516\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") "
Nov 03 20:14:26.943493 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:26.943445    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-credential-socket" pod "2b1c87e7-ecf0-438f-a3c4-93da53813516" (UID: "2b1c87e7-ecf0-438f-a3c4-93da53813516"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:14:26.943548 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:26.943508    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b1c87e7-ecf0-438f-a3c4-93da53813516-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1c87e7-ecf0-438f-a3c4-93da53813516-istiod-ca-cert\") pod \"2b1c87e7-ecf0-438f-a3c4-93da53813516\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") "
Nov 03 20:14:26.943614 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:26.943587    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-workload-socket\") pod \"2b1c87e7-ecf0-438f-a3c4-93da53813516\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") "
Nov 03 20:14:26.943672 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:26.943651    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-workload-certs\") pod \"2b1c87e7-ecf0-438f-a3c4-93da53813516\" (UID: \"2b1c87e7-ecf0-438f-a3c4-93da53813516\") "
Nov 03 20:14:26.943904 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:26.943876    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-workload-socket" pod "2b1c87e7-ecf0-438f-a3c4-93da53813516" (UID: "2b1c87e7-ecf0-438f-a3c4-93da53813516"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:14:26.944057 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:26.944032    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-workload-certs" pod "2b1c87e7-ecf0-438f-a3c4-93da53813516" (UID: "2b1c87e7-ecf0-438f-a3c4-93da53813516"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:14:26.944131 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:26.944113    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b1c87e7-ecf0-438f-a3c4-93da53813516-istiod-ca-cert" pod "2b1c87e7-ecf0-438f-a3c4-93da53813516" (UID: "2b1c87e7-ecf0-438f-a3c4-93da53813516"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:14:26.944721 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:26.944697    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:26.944815 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:26.944728    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1c87e7-ecf0-438f-a3c4-93da53813516-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:26.944815 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:26.944744    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:26.944815 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:26.944762    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:26.944815 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:26.944776    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:26.945361 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:26.945338    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-podinfo" pod "2b1c87e7-ecf0-438f-a3c4-93da53813516" (UID: "2b1c87e7-ecf0-438f-a3c4-93da53813516"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:14:26.945993 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:26.945969    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-envoy" pod "2b1c87e7-ecf0-438f-a3c4-93da53813516" (UID: "2b1c87e7-ecf0-438f-a3c4-93da53813516"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:14:26.946122 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:26.945994    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-token" pod "2b1c87e7-ecf0-438f-a3c4-93da53813516" (UID: "2b1c87e7-ecf0-438f-a3c4-93da53813516"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:14:26.946122 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:26.946082    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1c87e7-ecf0-438f-a3c4-93da53813516-kube-api-access-dpccm" pod "2b1c87e7-ecf0-438f-a3c4-93da53813516" (UID: "2b1c87e7-ecf0-438f-a3c4-93da53813516"). InnerVolumeSpecName "kube-api-access-dpccm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:14:27.046706 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.046666    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:27.046706 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.046704    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:27.046914 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.046719    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpccm\" (UniqueName: \"kubernetes.io/projected/2b1c87e7-ecf0-438f-a3c4-93da53813516-kube-api-access-dpccm\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:27.046914 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.046734    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1c87e7-ecf0-438f-a3c4-93da53813516-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:27.284975 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.284946    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 20:14:27.350458 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.350352    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c70d023-1700-42d0-964f-5573383d1355-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c70d023-1700-42d0-964f-5573383d1355-istio-podinfo\") pod \"3c70d023-1700-42d0-964f-5573383d1355\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") "
Nov 03 20:14:27.350458 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.350422    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c70d023-1700-42d0-964f-5573383d1355-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c70d023-1700-42d0-964f-5573383d1355-istiod-ca-cert\") pod \"3c70d023-1700-42d0-964f-5573383d1355\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") "
Nov 03 20:14:27.350707 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.350484    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-istio-data\") pod \"3c70d023-1700-42d0-964f-5573383d1355\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") "
Nov 03 20:14:27.350784 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.350746    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-workload-certs\") pod \"3c70d023-1700-42d0-964f-5573383d1355\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") "
Nov 03 20:14:27.350784 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.350756    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-istio-data" pod "3c70d023-1700-42d0-964f-5573383d1355" (UID: "3c70d023-1700-42d0-964f-5573383d1355"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:14:27.350904 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.350872    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c70d023-1700-42d0-964f-5573383d1355-kube-api-access-grzpd\" (UniqueName: \"kubernetes.io/projected/3c70d023-1700-42d0-964f-5573383d1355-kube-api-access-grzpd\") pod \"3c70d023-1700-42d0-964f-5573383d1355\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") "
Nov 03 20:14:27.350965 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.350908    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c70d023-1700-42d0-964f-5573383d1355-istiod-ca-cert" pod "3c70d023-1700-42d0-964f-5573383d1355" (UID: "3c70d023-1700-42d0-964f-5573383d1355"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:14:27.350965 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.350934    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c70d023-1700-42d0-964f-5573383d1355-istio-token\" (UniqueName: \"kubernetes.io/projected/3c70d023-1700-42d0-964f-5573383d1355-istio-token\") pod \"3c70d023-1700-42d0-964f-5573383d1355\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") "
Nov 03 20:14:27.351080 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.350989    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-istio-envoy\") pod \"3c70d023-1700-42d0-964f-5573383d1355\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") "
Nov 03 20:14:27.351080 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.351030    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-workload-socket\") pod \"3c70d023-1700-42d0-964f-5573383d1355\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") "
Nov 03 20:14:27.351080 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.351048    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-workload-certs" pod "3c70d023-1700-42d0-964f-5573383d1355" (UID: "3c70d023-1700-42d0-964f-5573383d1355"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:14:27.351326 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.351304    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-workload-socket" pod "3c70d023-1700-42d0-964f-5573383d1355" (UID: "3c70d023-1700-42d0-964f-5573383d1355"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:14:27.351704 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.351662    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-credential-socket\") pod \"3c70d023-1700-42d0-964f-5573383d1355\" (UID: \"3c70d023-1700-42d0-964f-5573383d1355\") "
Nov 03 20:14:27.352617 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.352590    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:27.352617 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.352624    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:27.352869 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.352640    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c70d023-1700-42d0-964f-5573383d1355-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:27.352869 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.352655    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:27.353042 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.352982    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-credential-socket" pod "3c70d023-1700-42d0-964f-5573383d1355" (UID: "3c70d023-1700-42d0-964f-5573383d1355"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:14:27.353391 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.353364    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c70d023-1700-42d0-964f-5573383d1355-istio-podinfo" pod "3c70d023-1700-42d0-964f-5573383d1355" (UID: "3c70d023-1700-42d0-964f-5573383d1355"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:14:27.353732 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.353701    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c70d023-1700-42d0-964f-5573383d1355-istio-token" pod "3c70d023-1700-42d0-964f-5573383d1355" (UID: "3c70d023-1700-42d0-964f-5573383d1355"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:14:27.353964 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.353932    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-istio-envoy" pod "3c70d023-1700-42d0-964f-5573383d1355" (UID: "3c70d023-1700-42d0-964f-5573383d1355"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:14:27.353964 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.353934    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c70d023-1700-42d0-964f-5573383d1355-kube-api-access-grzpd" pod "3c70d023-1700-42d0-964f-5573383d1355" (UID: "3c70d023-1700-42d0-964f-5573383d1355"). InnerVolumeSpecName "kube-api-access-grzpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:14:27.463277 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.458870    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grzpd\" (UniqueName: \"kubernetes.io/projected/3c70d023-1700-42d0-964f-5573383d1355-kube-api-access-grzpd\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:27.463277 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.458913    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c70d023-1700-42d0-964f-5573383d1355-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:27.463277 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.458929    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:27.463277 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.458942    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c70d023-1700-42d0-964f-5573383d1355-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:27.463277 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.458957    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c70d023-1700-42d0-964f-5573383d1355-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:14:27.627830 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.627744    5591 generic.go:356] "Generic (PLEG): container finished" podID="2b1c87e7-ecf0-438f-a3c4-93da53813516" containerID="968872f53ed39a1ad4244ad035af2045618c1bf05eedd7b4b81b430ffb55b8cf" exitCode=0
Nov 03 20:14:27.627830 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.627792    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"
Nov 03 20:14:27.628056 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.627831    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n" event={"ID":"2b1c87e7-ecf0-438f-a3c4-93da53813516","Type":"ContainerDied","Data":"968872f53ed39a1ad4244ad035af2045618c1bf05eedd7b4b81b430ffb55b8cf"}
Nov 03 20:14:27.628056 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.627886    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n" event={"ID":"2b1c87e7-ecf0-438f-a3c4-93da53813516","Type":"ContainerDied","Data":"b9b63d65772a2285af50eab871be450527504c0df71b3e5eac5fcc132b44b1d5"}
Nov 03 20:14:27.628056 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.627905    5591 scope.go:122] "RemoveContainer" containerID="968872f53ed39a1ad4244ad035af2045618c1bf05eedd7b4b81b430ffb55b8cf"
Nov 03 20:14:27.629596 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.629568    5591 generic.go:356] "Generic (PLEG): container finished" podID="3c70d023-1700-42d0-964f-5573383d1355" containerID="456c1f059ed2fdcef06f23f5cf106c7ca251c32337d4d7794f0d6c13e96268b9" exitCode=0
Nov 03 20:14:27.629722 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.629624    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"
Nov 03 20:14:27.629722 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.629653    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7" event={"ID":"3c70d023-1700-42d0-964f-5573383d1355","Type":"ContainerDied","Data":"456c1f059ed2fdcef06f23f5cf106c7ca251c32337d4d7794f0d6c13e96268b9"}
Nov 03 20:14:27.629722 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.629700    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7" event={"ID":"3c70d023-1700-42d0-964f-5573383d1355","Type":"ContainerDied","Data":"092f3cc7c453afc123104c3797c5d8d6a06ab4db4debde0b506a78e6233b3e19"}
Nov 03 20:14:27.641278 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.641089    5591 scope.go:122] "RemoveContainer" containerID="968872f53ed39a1ad4244ad035af2045618c1bf05eedd7b4b81b430ffb55b8cf"
Nov 03 20:14:27.641526 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:14:27.641485    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"968872f53ed39a1ad4244ad035af2045618c1bf05eedd7b4b81b430ffb55b8cf\": container with ID starting with 968872f53ed39a1ad4244ad035af2045618c1bf05eedd7b4b81b430ffb55b8cf not found: ID does not exist" containerID="968872f53ed39a1ad4244ad035af2045618c1bf05eedd7b4b81b430ffb55b8cf"
Nov 03 20:14:27.641591 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.641535    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"968872f53ed39a1ad4244ad035af2045618c1bf05eedd7b4b81b430ffb55b8cf"} err="failed to get container status \"968872f53ed39a1ad4244ad035af2045618c1bf05eedd7b4b81b430ffb55b8cf\": rpc error: code = NotFound desc = could not find container \"968872f53ed39a1ad4244ad035af2045618c1bf05eedd7b4b81b430ffb55b8cf\": container with ID starting with 968872f53ed39a1ad4244ad035af2045618c1bf05eedd7b4b81b430ffb55b8cf not found: ID does not exist"
Nov 03 20:14:27.641591 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.641551    5591 scope.go:122] "RemoveContainer" containerID="456c1f059ed2fdcef06f23f5cf106c7ca251c32337d4d7794f0d6c13e96268b9"
Nov 03 20:14:27.654697 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.654673    5591 scope.go:122] "RemoveContainer" containerID="456c1f059ed2fdcef06f23f5cf106c7ca251c32337d4d7794f0d6c13e96268b9"
Nov 03 20:14:27.655031 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:14:27.655010    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"456c1f059ed2fdcef06f23f5cf106c7ca251c32337d4d7794f0d6c13e96268b9\": container with ID starting with 456c1f059ed2fdcef06f23f5cf106c7ca251c32337d4d7794f0d6c13e96268b9 not found: ID does not exist" containerID="456c1f059ed2fdcef06f23f5cf106c7ca251c32337d4d7794f0d6c13e96268b9"
Nov 03 20:14:27.655084 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.655039    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"456c1f059ed2fdcef06f23f5cf106c7ca251c32337d4d7794f0d6c13e96268b9"} err="failed to get container status \"456c1f059ed2fdcef06f23f5cf106c7ca251c32337d4d7794f0d6c13e96268b9\": rpc error: code = NotFound desc = could not find container \"456c1f059ed2fdcef06f23f5cf106c7ca251c32337d4d7794f0d6c13e96268b9\": container with ID starting with 456c1f059ed2fdcef06f23f5cf106c7ca251c32337d4d7794f0d6c13e96268b9 not found: ID does not exist"
Nov 03 20:14:27.655486 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.655456    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"]
Nov 03 20:14:27.665314 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.665282    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-hj9k7"]
Nov 03 20:14:27.681272 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.681240    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"]
Nov 03 20:14:27.685377 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:27.685351    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-brv4n"]
Nov 03 20:14:28.122909 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.122871    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"]
Nov 03 20:14:28.124036 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.124012    5591 cpu_manager.go:414] "containerMap: removing container" podUID="8609c94b-9a4c-4335-8d07-4bfbf000b0e6" containerName="payload-processing"
Nov 03 20:14:28.124160 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.124037    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8609c94b-9a4c-4335-8d07-4bfbf000b0e6" containerName="payload-processing"
Nov 03 20:14:28.124160 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.124073    5591 cpu_manager.go:414] "containerMap: removing container" podUID="2b1c87e7-ecf0-438f-a3c4-93da53813516" containerName="istio-proxy"
Nov 03 20:14:28.124160 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.124080    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b1c87e7-ecf0-438f-a3c4-93da53813516" containerName="istio-proxy"
Nov 03 20:14:28.124160 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.124092    5591 cpu_manager.go:414] "containerMap: removing container" podUID="330b5625-e53f-4c73-93cb-98889bc2f0b8" containerName="payload-processing"
Nov 03 20:14:28.124160 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.124099    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="330b5625-e53f-4c73-93cb-98889bc2f0b8" containerName="payload-processing"
Nov 03 20:14:28.124160 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.124136    5591 cpu_manager.go:414] "containerMap: removing container" podUID="3c70d023-1700-42d0-964f-5573383d1355" containerName="istio-proxy"
Nov 03 20:14:28.124160 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.124143    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c70d023-1700-42d0-964f-5573383d1355" containerName="istio-proxy"
Nov 03 20:14:28.124440 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.124363    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8609c94b-9a4c-4335-8d07-4bfbf000b0e6" containerName="payload-processing"
Nov 03 20:14:28.124440 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.124379    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b1c87e7-ecf0-438f-a3c4-93da53813516" containerName="istio-proxy"
Nov 03 20:14:28.124440 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.124388    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="330b5625-e53f-4c73-93cb-98889bc2f0b8" containerName="payload-processing"
Nov 03 20:14:28.124440 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.124397    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c70d023-1700-42d0-964f-5573383d1355" containerName="istio-proxy"
Nov 03 20:14:28.130526 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.130503    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.141777 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.141710    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-eb69cb-openshift-default-dockercfg-w96q5\""
Nov 03 20:14:28.145808 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.145774    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"]
Nov 03 20:14:28.278537 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.278500    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/824d78ad-b227-4bdd-8245-6f37ef395666-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.278762 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.278547    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/824d78ad-b227-4bdd-8245-6f37ef395666-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.278762 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.278568    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.278762 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.278725    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/824d78ad-b227-4bdd-8245-6f37ef395666-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.278947 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.278916    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsj54\" (UniqueName: \"kubernetes.io/projected/824d78ad-b227-4bdd-8245-6f37ef395666-kube-api-access-rsj54\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.279002 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.278963    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.279069 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.279048    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.279126 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.279112    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.279260 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.279244    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.380501 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.380409    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.380675 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.380545    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.380675 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.380588    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/824d78ad-b227-4bdd-8245-6f37ef395666-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.380675 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.380622    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/824d78ad-b227-4bdd-8245-6f37ef395666-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.380795 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.380701    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.380835 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.380792    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/824d78ad-b227-4bdd-8245-6f37ef395666-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.380899 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.380885    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.381024 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.380987    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsj54\" (UniqueName: \"kubernetes.io/projected/824d78ad-b227-4bdd-8245-6f37ef395666-kube-api-access-rsj54\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.381149 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.381053    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.381149 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.381108    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.381326 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.381051    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.381562 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.381364    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.381562 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.381419    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.381562 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.381550    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/824d78ad-b227-4bdd-8245-6f37ef395666-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.383198 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.383176    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.383352 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.383334    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/824d78ad-b227-4bdd-8245-6f37ef395666-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.392920 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.392892    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/824d78ad-b227-4bdd-8245-6f37ef395666-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.393141 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.393112    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsj54\" (UniqueName: \"kubernetes.io/projected/824d78ad-b227-4bdd-8245-6f37ef395666-kube-api-access-rsj54\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.449952 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.449912    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:28.736292 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:14:28.736166    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:14:28.770143 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.770107    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"]
Nov 03 20:14:28.774828 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.774762    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:14:28.774974 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.774876    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:14:28.774974 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:28.774916    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:14:29.355652 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:29.355614    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b1c87e7-ecf0-438f-a3c4-93da53813516" path="/var/lib/kubelet/pods/2b1c87e7-ecf0-438f-a3c4-93da53813516/volumes"
Nov 03 20:14:29.356462 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:29.356428    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c70d023-1700-42d0-964f-5573383d1355" path="/var/lib/kubelet/pods/3c70d023-1700-42d0-964f-5573383d1355/volumes"
Nov 03 20:14:29.553062 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:29.553029    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:14:29.655637 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:29.655550    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg" event={"ID":"824d78ad-b227-4bdd-8245-6f37ef395666","Type":"ContainerStarted","Data":"fd411a736a6c9774b699a67fff682aa01b3a17a335e7209c49ea50f12376341b"}
Nov 03 20:14:29.655637 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:29.655599    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg" event={"ID":"824d78ad-b227-4bdd-8245-6f37ef395666","Type":"ContainerStarted","Data":"90ed241708fdd35a6a170a80b906648cf0bc89c7fc432db72e0650f17f232afd"}
Nov 03 20:14:29.707225 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:29.706971    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg" podStartSLOduration=1.70695067 podStartE2EDuration="1.70695067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:29.703359811 +0000 UTC m=+2218.475652829" watchObservedRunningTime="2026-11-03 20:14:29.70695067 +0000 UTC m=+2218.479243690"
Nov 03 20:14:30.450737 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:30.450702    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:30.452456 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:30.452427    5591 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body=
Nov 03 20:14:30.452603 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:30.452496    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg" podUID="824d78ad-b227-4bdd-8245-6f37ef395666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused"
Nov 03 20:14:31.495251 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:31.480184    5591 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body=
Nov 03 20:14:31.495251 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:31.480281    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg" podUID="824d78ad-b227-4bdd-8245-6f37ef395666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused"
Nov 03 20:14:32.452238 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:32.451640    5591 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body=
Nov 03 20:14:32.452238 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:32.451704    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg" podUID="824d78ad-b227-4bdd-8245-6f37ef395666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused"
Nov 03 20:14:33.451337 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:33.451298    5591 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body=
Nov 03 20:14:33.451744 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:33.451356    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg" podUID="824d78ad-b227-4bdd-8245-6f37ef395666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused"
Nov 03 20:14:34.450222 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:34.450181    5591 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body=
Nov 03 20:14:34.450446 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:34.450285    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg" podUID="824d78ad-b227-4bdd-8245-6f37ef395666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused"
Nov 03 20:14:35.450758 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:35.450716    5591 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body=
Nov 03 20:14:35.451327 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:35.450797    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg" podUID="824d78ad-b227-4bdd-8245-6f37ef395666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused"
Nov 03 20:14:36.457634 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:36.457600    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:36.458173 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:36.458082    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:36.459177 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:36.459153    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:14:55.275053 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:55.275015    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62"]
Nov 03 20:14:55.288651 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:55.286550    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62"
Nov 03 20:14:55.294072 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:55.294023    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"agc-cred-8b838a662572\""
Nov 03 20:14:55.294732 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:55.294710    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"agc-cred-f5fb49f604b6\""
Nov 03 20:14:55.297430 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:55.294931    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"payload-processing-external-model-plugins-e2e-extmod-b-4c2a28\""
Nov 03 20:14:55.297430 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:55.295023    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"routing-overlay\""
Nov 03 20:14:55.297430 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:55.295063    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"payload-processing-external-model-e2e--088821c0-dockercfg-mqh9q\""
Nov 03 20:14:55.297430 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:55.295093    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"openshift-service-ca.crt\""
Nov 03 20:14:55.297430 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:55.294847    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4c2a28\"/\"kube-root-ca.crt\""
Nov 03 20:14:55.318891 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:55.318852    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62\" (UID: \"6e4fb2e7-9e81-4958-ac0e-6204fae57dc9\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62"
Nov 03 20:14:55.319193 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:55.319169    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62\" (UID: \"6e4fb2e7-9e81-4958-ac0e-6204fae57dc9\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62"
Nov 03 20:14:55.319633 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:55.319606    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62\" (UID: \"6e4fb2e7-9e81-4958-ac0e-6204fae57dc9\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62"
Nov 03 20:14:55.319889 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:55.319866    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfs7s\" (UniqueName: \"kubernetes.io/projected/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-kube-api-access-wfs7s\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62\" (UID: \"6e4fb2e7-9e81-4958-ac0e-6204fae57dc9\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62"
Nov 03 20:14:55.330078 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:55.330022    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62"]
Nov 03 20:14:55.422195 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:55.422158    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62\" (UID: \"6e4fb2e7-9e81-4958-ac0e-6204fae57dc9\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62"
Nov 03 20:14:55.422397 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:55.422281    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfs7s\" (UniqueName: \"kubernetes.io/projected/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-kube-api-access-wfs7s\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62\" (UID: \"6e4fb2e7-9e81-4958-ac0e-6204fae57dc9\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62"
Nov 03 20:14:55.422507 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:55.422488    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62\" (UID: \"6e4fb2e7-9e81-4958-ac0e-6204fae57dc9\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62"
Nov 03 20:14:55.422600 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:55.422526    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62\" (UID: \"6e4fb2e7-9e81-4958-ac0e-6204fae57dc9\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62"
Nov 03 20:14:55.423044 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:55.423023    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62\" (UID: \"6e4fb2e7-9e81-4958-ac0e-6204fae57dc9\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62"
Nov 03 20:14:55.423121 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:55.423099    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62\" (UID: \"6e4fb2e7-9e81-4958-ac0e-6204fae57dc9\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62"
Nov 03 20:14:55.425242 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:55.425220    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62\" (UID: \"6e4fb2e7-9e81-4958-ac0e-6204fae57dc9\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62"
Nov 03 20:14:55.434590 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:55.434561    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfs7s\" (UniqueName: \"kubernetes.io/projected/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-kube-api-access-wfs7s\") pod \"payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62\" (UID: \"6e4fb2e7-9e81-4958-ac0e-6204fae57dc9\") " pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62"
Nov 03 20:14:55.643372 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:55.643279    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62"
Nov 03 20:14:55.934916 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:55.934884    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62"]
Nov 03 20:14:55.935640 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:14:55.935611    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e4fb2e7_9e81_4958_ac0e_6204fae57dc9.slice/crio-eec682e48b7051b22b09b1a76b9cb7052b9f6874bfd55d8c0dfca232fa3b43d4 WatchSource:0}: Error finding container eec682e48b7051b22b09b1a76b9cb7052b9f6874bfd55d8c0dfca232fa3b43d4: Status 404 returned error can't find the container with id eec682e48b7051b22b09b1a76b9cb7052b9f6874bfd55d8c0dfca232fa3b43d4
Nov 03 20:14:56.069943 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:56.069904    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62" event={"ID":"6e4fb2e7-9e81-4958-ac0e-6204fae57dc9","Type":"ContainerStarted","Data":"02562140506cf26044cef48f452a820cb59207d1b1ae064f0d34e93fdb1fdd79"}
Nov 03 20:14:56.070167 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:56.069971    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62" event={"ID":"6e4fb2e7-9e81-4958-ac0e-6204fae57dc9","Type":"ContainerStarted","Data":"eec682e48b7051b22b09b1a76b9cb7052b9f6874bfd55d8c0dfca232fa3b43d4"}
Nov 03 20:14:56.070167 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:56.069998    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62"
Nov 03 20:14:56.091651 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:56.091581    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62" podStartSLOduration=1.091564822 podStartE2EDuration="1.091564822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:56.088883653 +0000 UTC m=+2244.861176671" watchObservedRunningTime="2026-11-03 20:14:56.091564822 +0000 UTC m=+2244.863857840"
Nov 03 20:14:57.401422 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:57.401385    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6"]
Nov 03 20:14:57.406719 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:57.406693    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6"
Nov 03 20:14:57.414880 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:57.414849    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"kube-root-ca.crt\""
Nov 03 20:14:57.414880 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:57.414849    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"agc-cred-ba07612cde79\""
Nov 03 20:14:57.415163 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:57.414894    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"e2e-cfl-8de789-provider-cred\""
Nov 03 20:14:57.415163 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:57.414912    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"payload-processing-external-model-plugins-e2e-extmod-a-98529d\""
Nov 03 20:14:57.415163 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:57.414894    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"routing-overlay\""
Nov 03 20:14:57.415163 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:57.414956    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"openshift-service-ca.crt\""
Nov 03 20:14:57.415163 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:57.414956    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-98529d\"/\"payload-processing-external-model-e2e--c4b4e61e-dockercfg-8sdl4\""
Nov 03 20:14:57.418702 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:57.418381    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6"]
Nov 03 20:14:57.517324 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:57.517288    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b177f2e-45af-4fd7-b033-13baeac28f49-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6\" (UID: \"3b177f2e-45af-4fd7-b033-13baeac28f49\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6"
Nov 03 20:14:57.517324 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:57.517325    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbqn6\" (UniqueName: \"kubernetes.io/projected/3b177f2e-45af-4fd7-b033-13baeac28f49-kube-api-access-fbqn6\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6\" (UID: \"3b177f2e-45af-4fd7-b033-13baeac28f49\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6"
Nov 03 20:14:57.517592 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:57.517513    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3b177f2e-45af-4fd7-b033-13baeac28f49-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6\" (UID: \"3b177f2e-45af-4fd7-b033-13baeac28f49\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6"
Nov 03 20:14:57.517670 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:57.517645    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3b177f2e-45af-4fd7-b033-13baeac28f49-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6\" (UID: \"3b177f2e-45af-4fd7-b033-13baeac28f49\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6"
Nov 03 20:14:57.619179 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:57.619135    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3b177f2e-45af-4fd7-b033-13baeac28f49-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6\" (UID: \"3b177f2e-45af-4fd7-b033-13baeac28f49\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6"
Nov 03 20:14:57.619423 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:57.619286    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3b177f2e-45af-4fd7-b033-13baeac28f49-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6\" (UID: \"3b177f2e-45af-4fd7-b033-13baeac28f49\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6"
Nov 03 20:14:57.619534 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:57.619515    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b177f2e-45af-4fd7-b033-13baeac28f49-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6\" (UID: \"3b177f2e-45af-4fd7-b033-13baeac28f49\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6"
Nov 03 20:14:57.619591 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:57.619550    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbqn6\" (UniqueName: \"kubernetes.io/projected/3b177f2e-45af-4fd7-b033-13baeac28f49-kube-api-access-fbqn6\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6\" (UID: \"3b177f2e-45af-4fd7-b033-13baeac28f49\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6"
Nov 03 20:14:57.620021 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:57.619997    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3b177f2e-45af-4fd7-b033-13baeac28f49-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6\" (UID: \"3b177f2e-45af-4fd7-b033-13baeac28f49\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6"
Nov 03 20:14:57.620212 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:57.620179    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b177f2e-45af-4fd7-b033-13baeac28f49-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6\" (UID: \"3b177f2e-45af-4fd7-b033-13baeac28f49\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6"
Nov 03 20:14:57.622363 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:57.622340    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3b177f2e-45af-4fd7-b033-13baeac28f49-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6\" (UID: \"3b177f2e-45af-4fd7-b033-13baeac28f49\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6"
Nov 03 20:14:57.632368 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:57.632335    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbqn6\" (UniqueName: \"kubernetes.io/projected/3b177f2e-45af-4fd7-b033-13baeac28f49-kube-api-access-fbqn6\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6\" (UID: \"3b177f2e-45af-4fd7-b033-13baeac28f49\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6"
Nov 03 20:14:57.727389 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:57.727299    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6"
Nov 03 20:14:58.097782 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.097747    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6"]
Nov 03 20:14:58.097916 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:14:58.097818    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b177f2e_45af_4fd7_b033_13baeac28f49.slice/crio-3a1fcc04ee54e2e74e89ec076bb674e166bf6da36cf3942e25112cb90a1c3db1 WatchSource:0}: Error finding container 3a1fcc04ee54e2e74e89ec076bb674e166bf6da36cf3942e25112cb90a1c3db1: Status 404 returned error can't find the container with id 3a1fcc04ee54e2e74e89ec076bb674e166bf6da36cf3942e25112cb90a1c3db1
Nov 03 20:14:58.131177 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.131138    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6" event={"ID":"3b177f2e-45af-4fd7-b033-13baeac28f49","Type":"ContainerStarted","Data":"3a1fcc04ee54e2e74e89ec076bb674e166bf6da36cf3942e25112cb90a1c3db1"}
Nov 03 20:14:58.166269 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.166239    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"]
Nov 03 20:14:58.173087 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.173062    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.186730 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.186699    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"]
Nov 03 20:14:58.334834 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.334789    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/120358c4-5509-4cc4-8522-647ddbde2d90-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.335005 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.334970    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.335005 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.334999    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/120358c4-5509-4cc4-8522-647ddbde2d90-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.335258 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.335229    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvdtp\" (UniqueName: \"kubernetes.io/projected/120358c4-5509-4cc4-8522-647ddbde2d90-kube-api-access-pvdtp\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.335422 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.335399    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/120358c4-5509-4cc4-8522-647ddbde2d90-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.335507 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.335459    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.335571 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.335555    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.335635 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.335600    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.335690 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.335672    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.436838 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.436738    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/120358c4-5509-4cc4-8522-647ddbde2d90-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.436838 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.436825    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.437427 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.436865    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.437427 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.436918    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.437427 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.436949    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.437427 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.437097    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/120358c4-5509-4cc4-8522-647ddbde2d90-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.437427 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.437174    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.437427 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.437196    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/120358c4-5509-4cc4-8522-647ddbde2d90-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.437427 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.437411    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-credential-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.437833 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.437445    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvdtp\" (UniqueName: \"kubernetes.io/projected/120358c4-5509-4cc4-8522-647ddbde2d90-kube-api-access-pvdtp\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.437833 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.437678    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-workload-socket\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.438248 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.438193    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/120358c4-5509-4cc4-8522-647ddbde2d90-istiod-ca-cert\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.438248 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.438228    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-istio-data\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.438394 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.438304    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-workload-certs\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.439780 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.439760    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/120358c4-5509-4cc4-8522-647ddbde2d90-istio-podinfo\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.440045 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.440025    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-istio-envoy\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.451375 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.451344    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/120358c4-5509-4cc4-8522-647ddbde2d90-istio-token\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.451633 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.451604    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvdtp\" (UniqueName: \"kubernetes.io/projected/120358c4-5509-4cc4-8522-647ddbde2d90-kube-api-access-pvdtp\") pod \"e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") " pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.494727 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.494691    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:58.738938 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:14:58.738834    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:14:58.897224 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.897175    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"]
Nov 03 20:14:58.898096 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:14:58.898064    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod120358c4_5509_4cc4_8522_647ddbde2d90.slice/crio-ff5d2971dd84a8e0d9ac1b56aaa796f89a56d707bfa7172d441ae3305bcf5f53 WatchSource:0}: Error finding container ff5d2971dd84a8e0d9ac1b56aaa796f89a56d707bfa7172d441ae3305bcf5f53: Status 404 returned error can't find the container with id ff5d2971dd84a8e0d9ac1b56aaa796f89a56d707bfa7172d441ae3305bcf5f53
Nov 03 20:14:58.901724 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.901688    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:14:58.901817 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.901761    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:14:58.901817 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:58.901791    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:14:59.141849 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:59.141817    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt" event={"ID":"120358c4-5509-4cc4-8522-647ddbde2d90","Type":"ContainerStarted","Data":"07f19dbfe7cf353f847f514c30cb11bb1df4e5a2501fe576cab388803dddee5d"}
Nov 03 20:14:59.142073 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:59.141875    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt" event={"ID":"120358c4-5509-4cc4-8522-647ddbde2d90","Type":"ContainerStarted","Data":"ff5d2971dd84a8e0d9ac1b56aaa796f89a56d707bfa7172d441ae3305bcf5f53"}
Nov 03 20:14:59.143856 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:59.143819    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6" event={"ID":"3b177f2e-45af-4fd7-b033-13baeac28f49","Type":"ContainerStarted","Data":"e30d5d05f86bded49d6ec609ebabbcb522c3657f1fa1679f8da060b09ce073c0"}
Nov 03 20:14:59.144001 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:59.143910    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6"
Nov 03 20:14:59.171391 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:59.171330    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt" podStartSLOduration=1.171311131 podStartE2EDuration="1.171311131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:59.165918282 +0000 UTC m=+2247.938211301" watchObservedRunningTime="2026-11-03 20:14:59.171311131 +0000 UTC m=+2247.943604153"
Nov 03 20:14:59.190948 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:59.190886    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6" podStartSLOduration=2.190866225 podStartE2EDuration="2.190866225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:59.187926766 +0000 UTC m=+2247.960219783" watchObservedRunningTime="2026-11-03 20:14:59.190866225 +0000 UTC m=+2247.963159241"
Nov 03 20:14:59.495672 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:59.495633    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:14:59.497940 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:59.497910    5591 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 03 20:14:59.498111 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:14:59.497964    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt" podUID="120358c4-5509-4cc4-8522-647ddbde2d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 03 20:15:00.241273 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.234187    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp"]
Nov 03 20:15:00.256689 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.256654    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb"]
Nov 03 20:15:00.256932 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.256907    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp"
Nov 03 20:15:00.267274 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.267064    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\""
Nov 03 20:15:00.272468 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.271437    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj"]
Nov 03 20:15:00.275589 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.274014    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb"
Nov 03 20:15:00.279094 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.277253    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfvc2\" (UniqueName: \"kubernetes.io/projected/08e20158-6af2-4cec-ba28-74866c9a2731-kube-api-access-bfvc2\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp\" (UID: \"08e20158-6af2-4cec-ba28-74866c9a2731\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp"
Nov 03 20:15:00.279094 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.277528    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08e20158-6af2-4cec-ba28-74866c9a2731-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp\" (UID: \"08e20158-6af2-4cec-ba28-74866c9a2731\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp"
Nov 03 20:15:00.446572 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.446533    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf"]
Nov 03 20:15:00.447666 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.447631    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfm7c\" (UniqueName: \"kubernetes.io/projected/59f8a1f2-c5c6-4d93-8477-60ddff9ea8df-kube-api-access-mfm7c\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb\" (UID: \"59f8a1f2-c5c6-4d93-8477-60ddff9ea8df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb"
Nov 03 20:15:00.447829 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.447733    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59f8a1f2-c5c6-4d93-8477-60ddff9ea8df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb\" (UID: \"59f8a1f2-c5c6-4d93-8477-60ddff9ea8df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb"
Nov 03 20:15:00.447899 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.447852    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfvc2\" (UniqueName: \"kubernetes.io/projected/08e20158-6af2-4cec-ba28-74866c9a2731-kube-api-access-bfvc2\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp\" (UID: \"08e20158-6af2-4cec-ba28-74866c9a2731\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp"
Nov 03 20:15:00.448040 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.448027    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08e20158-6af2-4cec-ba28-74866c9a2731-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp\" (UID: \"08e20158-6af2-4cec-ba28-74866c9a2731\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp"
Nov 03 20:15:00.448936 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.448902    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08e20158-6af2-4cec-ba28-74866c9a2731-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp\" (UID: \"08e20158-6af2-4cec-ba28-74866c9a2731\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp"
Nov 03 20:15:00.449134 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.449114    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj"
Nov 03 20:15:00.453302 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.453277    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp"]
Nov 03 20:15:00.453508 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.453492    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb"]
Nov 03 20:15:00.453639 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.453627    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf"]
Nov 03 20:15:00.453827 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.453813    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj"]
Nov 03 20:15:00.454588 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.453400    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf"
Nov 03 20:15:00.464466 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.464422    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfvc2\" (UniqueName: \"kubernetes.io/projected/08e20158-6af2-4cec-ba28-74866c9a2731-kube-api-access-bfvc2\") pod \"maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp\" (UID: \"08e20158-6af2-4cec-ba28-74866c9a2731\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp"
Nov 03 20:15:00.516132 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.516095    5591 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 03 20:15:00.516658 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.516169    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt" podUID="120358c4-5509-4cc4-8522-647ddbde2d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 03 20:15:00.549592 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.549553    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59f8a1f2-c5c6-4d93-8477-60ddff9ea8df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb\" (UID: \"59f8a1f2-c5c6-4d93-8477-60ddff9ea8df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb"
Nov 03 20:15:00.549814 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.549782    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f1d45f5-4ee7-427b-a26e-e14568a5cf51-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf\" (UID: \"4f1d45f5-4ee7-427b-a26e-e14568a5cf51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf"
Nov 03 20:15:00.549901 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.549887    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tdsz\" (UniqueName: \"kubernetes.io/projected/4f1d45f5-4ee7-427b-a26e-e14568a5cf51-kube-api-access-4tdsz\") pod \"maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf\" (UID: \"4f1d45f5-4ee7-427b-a26e-e14568a5cf51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf"
Nov 03 20:15:00.550587 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.550218    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9344a596-282c-4719-a323-5771291e1cdb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj\" (UID: \"9344a596-282c-4719-a323-5771291e1cdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj"
Nov 03 20:15:00.550587 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.550300    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4fd5\" (UniqueName: \"kubernetes.io/projected/9344a596-282c-4719-a323-5771291e1cdb-kube-api-access-b4fd5\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj\" (UID: \"9344a596-282c-4719-a323-5771291e1cdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj"
Nov 03 20:15:00.550587 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.550345    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59f8a1f2-c5c6-4d93-8477-60ddff9ea8df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb\" (UID: \"59f8a1f2-c5c6-4d93-8477-60ddff9ea8df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb"
Nov 03 20:15:00.550587 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.550467    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfm7c\" (UniqueName: \"kubernetes.io/projected/59f8a1f2-c5c6-4d93-8477-60ddff9ea8df-kube-api-access-mfm7c\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb\" (UID: \"59f8a1f2-c5c6-4d93-8477-60ddff9ea8df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb"
Nov 03 20:15:00.563412 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.563371    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfm7c\" (UniqueName: \"kubernetes.io/projected/59f8a1f2-c5c6-4d93-8477-60ddff9ea8df-kube-api-access-mfm7c\") pod \"maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb\" (UID: \"59f8a1f2-c5c6-4d93-8477-60ddff9ea8df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb"
Nov 03 20:15:00.652017 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.651980    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f1d45f5-4ee7-427b-a26e-e14568a5cf51-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf\" (UID: \"4f1d45f5-4ee7-427b-a26e-e14568a5cf51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf"
Nov 03 20:15:00.652177 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.652057    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tdsz\" (UniqueName: \"kubernetes.io/projected/4f1d45f5-4ee7-427b-a26e-e14568a5cf51-kube-api-access-4tdsz\") pod \"maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf\" (UID: \"4f1d45f5-4ee7-427b-a26e-e14568a5cf51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf"
Nov 03 20:15:00.652177 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.652164    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9344a596-282c-4719-a323-5771291e1cdb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj\" (UID: \"9344a596-282c-4719-a323-5771291e1cdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj"
Nov 03 20:15:00.652318 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.652196    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4fd5\" (UniqueName: \"kubernetes.io/projected/9344a596-282c-4719-a323-5771291e1cdb-kube-api-access-b4fd5\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj\" (UID: \"9344a596-282c-4719-a323-5771291e1cdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj"
Nov 03 20:15:00.652705 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.652681    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f1d45f5-4ee7-427b-a26e-e14568a5cf51-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf\" (UID: \"4f1d45f5-4ee7-427b-a26e-e14568a5cf51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf"
Nov 03 20:15:00.652854 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.652832    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9344a596-282c-4719-a323-5771291e1cdb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj\" (UID: \"9344a596-282c-4719-a323-5771291e1cdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj"
Nov 03 20:15:00.665516 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.665482    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4fd5\" (UniqueName: \"kubernetes.io/projected/9344a596-282c-4719-a323-5771291e1cdb-kube-api-access-b4fd5\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj\" (UID: \"9344a596-282c-4719-a323-5771291e1cdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj"
Nov 03 20:15:00.665663 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.665517    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tdsz\" (UniqueName: \"kubernetes.io/projected/4f1d45f5-4ee7-427b-a26e-e14568a5cf51-kube-api-access-4tdsz\") pod \"maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf\" (UID: \"4f1d45f5-4ee7-427b-a26e-e14568a5cf51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf"
Nov 03 20:15:00.745349 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.745308    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb"
Nov 03 20:15:00.758053 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.758018    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp"
Nov 03 20:15:00.836890 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.836585    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj"
Nov 03 20:15:00.844522 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:00.842754    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf"
Nov 03 20:15:01.079496 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:01.078460    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb"]
Nov 03 20:15:01.112089 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:15:01.111479    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f8a1f2_c5c6_4d93_8477_60ddff9ea8df.slice/crio-29faccaeef4a90e6f79089047ce9d93646650152f593e82cc4d35b2964f3acd1 WatchSource:0}: Error finding container 29faccaeef4a90e6f79089047ce9d93646650152f593e82cc4d35b2964f3acd1: Status 404 returned error can't find the container with id 29faccaeef4a90e6f79089047ce9d93646650152f593e82cc4d35b2964f3acd1
Nov 03 20:15:01.227708 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:01.227668    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb" event={"ID":"59f8a1f2-c5c6-4d93-8477-60ddff9ea8df","Type":"ContainerStarted","Data":"29faccaeef4a90e6f79089047ce9d93646650152f593e82cc4d35b2964f3acd1"}
Nov 03 20:15:01.313228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:01.313164    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp"]
Nov 03 20:15:01.313595 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:15:01.313573    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08e20158_6af2_4cec_ba28_74866c9a2731.slice/crio-9caee87bc4139271e94890e5a80df1d5b198df75aa4df6023c52a421768a6109 WatchSource:0}: Error finding container 9caee87bc4139271e94890e5a80df1d5b198df75aa4df6023c52a421768a6109: Status 404 returned error can't find the container with id 9caee87bc4139271e94890e5a80df1d5b198df75aa4df6023c52a421768a6109
Nov 03 20:15:01.405402 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:01.405358    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj"]
Nov 03 20:15:01.438468 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:01.438437    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf"]
Nov 03 20:15:01.510230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:01.507454    5591 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 03 20:15:01.510230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:01.507520    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt" podUID="120358c4-5509-4cc4-8522-647ddbde2d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 03 20:15:02.240691 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:02.240590    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf" event={"ID":"4f1d45f5-4ee7-427b-a26e-e14568a5cf51","Type":"ContainerStarted","Data":"ecb2ff8b500b3be01dba68f716ad3422a2ebc9a3a13f599d8532a747a0ee1c4d"}
Nov 03 20:15:02.242682 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:02.242647    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp" event={"ID":"08e20158-6af2-4cec-ba28-74866c9a2731","Type":"ContainerStarted","Data":"4e7cd6ff438c58ad0a8d6b9bcbf34e9d21baac2acf0e609a7b88daf026dd6906"}
Nov 03 20:15:02.242906 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:02.242707    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp" event={"ID":"08e20158-6af2-4cec-ba28-74866c9a2731","Type":"ContainerStarted","Data":"9caee87bc4139271e94890e5a80df1d5b198df75aa4df6023c52a421768a6109"}
Nov 03 20:15:02.244162 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:02.244134    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj" event={"ID":"9344a596-282c-4719-a323-5771291e1cdb","Type":"ContainerStarted","Data":"ae849fca72c83c2c6c8ccc7078daebeca9ccd9c759c03d4bf4cd06f48ab92421"}
Nov 03 20:15:02.245718 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:02.245693    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb" event={"ID":"59f8a1f2-c5c6-4d93-8477-60ddff9ea8df","Type":"ContainerStarted","Data":"57392c03f4e12a2a185fda40c07eb9e917cc39065f1bd96fdaafe4d49f0919d7"}
Nov 03 20:15:02.262598 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:02.262532    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp" podStartSLOduration=2.262510988 podStartE2EDuration="2.262510988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:02.259908934 +0000 UTC m=+2251.032201953" watchObservedRunningTime="2026-11-03 20:15:02.262510988 +0000 UTC m=+2251.034804006"
Nov 03 20:15:02.499315 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:02.498650    5591 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 03 20:15:02.499315 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:02.498720    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt" podUID="120358c4-5509-4cc4-8522-647ddbde2d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 03 20:15:03.259540 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:03.259439    5591 generic.go:356] "Generic (PLEG): container finished" podID="08e20158-6af2-4cec-ba28-74866c9a2731" containerID="4e7cd6ff438c58ad0a8d6b9bcbf34e9d21baac2acf0e609a7b88daf026dd6906" exitCode=60
Nov 03 20:15:03.260038 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:03.259567    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp" event={"ID":"08e20158-6af2-4cec-ba28-74866c9a2731","Type":"ContainerDied","Data":"4e7cd6ff438c58ad0a8d6b9bcbf34e9d21baac2acf0e609a7b88daf026dd6906"}
Nov 03 20:15:03.260038 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:03.259987    5591 scope.go:122] "RemoveContainer" containerID="4e7cd6ff438c58ad0a8d6b9bcbf34e9d21baac2acf0e609a7b88daf026dd6906"
Nov 03 20:15:03.262975 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:03.262945    5591 generic.go:356] "Generic (PLEG): container finished" podID="9344a596-282c-4719-a323-5771291e1cdb" containerID="1be50c7102b5064bff329ea64309d40a75cc11389cfdb08811e4e6cde3fd7ed0" exitCode=60
Nov 03 20:15:03.263929 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:03.263000    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj" event={"ID":"9344a596-282c-4719-a323-5771291e1cdb","Type":"ContainerDied","Data":"1be50c7102b5064bff329ea64309d40a75cc11389cfdb08811e4e6cde3fd7ed0"}
Nov 03 20:15:03.265511 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:03.265489    5591 scope.go:122] "RemoveContainer" containerID="1be50c7102b5064bff329ea64309d40a75cc11389cfdb08811e4e6cde3fd7ed0"
Nov 03 20:15:03.266686 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:03.266480    5591 generic.go:356] "Generic (PLEG): container finished" podID="59f8a1f2-c5c6-4d93-8477-60ddff9ea8df" containerID="57392c03f4e12a2a185fda40c07eb9e917cc39065f1bd96fdaafe4d49f0919d7" exitCode=60
Nov 03 20:15:03.266971 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:03.266947    5591 scope.go:122] "RemoveContainer" containerID="57392c03f4e12a2a185fda40c07eb9e917cc39065f1bd96fdaafe4d49f0919d7"
Nov 03 20:15:03.267182 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:03.267114    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb" event={"ID":"59f8a1f2-c5c6-4d93-8477-60ddff9ea8df","Type":"ContainerDied","Data":"57392c03f4e12a2a185fda40c07eb9e917cc39065f1bd96fdaafe4d49f0919d7"}
Nov 03 20:15:03.269617 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:03.269593    5591 generic.go:356] "Generic (PLEG): container finished" podID="4f1d45f5-4ee7-427b-a26e-e14568a5cf51" containerID="77746a30efcb482e99bf270fe835da29043cb26ab5efd605151f07eecf66a3e7" exitCode=60
Nov 03 20:15:03.269718 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:03.269674    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf" event={"ID":"4f1d45f5-4ee7-427b-a26e-e14568a5cf51","Type":"ContainerDied","Data":"77746a30efcb482e99bf270fe835da29043cb26ab5efd605151f07eecf66a3e7"}
Nov 03 20:15:03.270142 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:03.270120    5591 scope.go:122] "RemoveContainer" containerID="77746a30efcb482e99bf270fe835da29043cb26ab5efd605151f07eecf66a3e7"
Nov 03 20:15:03.322273 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:03.319568    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb" podStartSLOduration=3.3195291239999998 podStartE2EDuration="3.319529124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:02.311097775 +0000 UTC m=+2251.083390792" watchObservedRunningTime="2026-11-03 20:15:03.319529124 +0000 UTC m=+2252.091822154"
Nov 03 20:15:03.563505 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:03.563404    5591 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 03 20:15:03.563718 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:03.563614    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt" podUID="120358c4-5509-4cc4-8522-647ddbde2d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 03 20:15:04.311913 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:04.311876    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp" event={"ID":"08e20158-6af2-4cec-ba28-74866c9a2731","Type":"ContainerStarted","Data":"011b2e26ccf39c9305fed407aed46712a280e4b6b9a6f09c689e7766dfb42263"}
Nov 03 20:15:04.314427 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:04.314398    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj" event={"ID":"9344a596-282c-4719-a323-5771291e1cdb","Type":"ContainerStarted","Data":"aabbea3e7b283052e0853ae40ac285a4cf115694081dab051f85def6f6b41071"}
Nov 03 20:15:04.352300 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:04.352240    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj" podStartSLOduration=4.352219207 podStartE2EDuration="4.352219207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:04.349857413 +0000 UTC m=+2253.122150426" watchObservedRunningTime="2026-11-03 20:15:04.352219207 +0000 UTC m=+2253.124512227"
Nov 03 20:15:04.513232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:04.506586    5591 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 03 20:15:04.513232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:04.506682    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt" podUID="120358c4-5509-4cc4-8522-647ddbde2d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 03 20:15:05.347966 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:05.347870    5591 generic.go:356] "Generic (PLEG): container finished" podID="4f1d45f5-4ee7-427b-a26e-e14568a5cf51" containerID="8d60ecb989c2c45e3d2a2aaa398350364227630ce949ecbdb35d37f56113af36" exitCode=60
Nov 03 20:15:05.348706 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:05.348245    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf" event={"ID":"4f1d45f5-4ee7-427b-a26e-e14568a5cf51","Type":"ContainerDied","Data":"8d60ecb989c2c45e3d2a2aaa398350364227630ce949ecbdb35d37f56113af36"}
Nov 03 20:15:05.348706 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:05.348393    5591 scope.go:122] "RemoveContainer" containerID="77746a30efcb482e99bf270fe835da29043cb26ab5efd605151f07eecf66a3e7"
Nov 03 20:15:05.349305 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:05.349092    5591 scope.go:122] "RemoveContainer" containerID="8d60ecb989c2c45e3d2a2aaa398350364227630ce949ecbdb35d37f56113af36"
Nov 03 20:15:05.349633 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:15:05.349605    5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf_odh-ai-gateway-infra(4f1d45f5-4ee7-427b-a26e-e14568a5cf51)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf" podUID="4f1d45f5-4ee7-427b-a26e-e14568a5cf51"
Nov 03 20:15:05.358849 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:05.358233    5591 generic.go:356] "Generic (PLEG): container finished" podID="08e20158-6af2-4cec-ba28-74866c9a2731" containerID="011b2e26ccf39c9305fed407aed46712a280e4b6b9a6f09c689e7766dfb42263" exitCode=60
Nov 03 20:15:05.358849 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:05.358774    5591 scope.go:122] "RemoveContainer" containerID="011b2e26ccf39c9305fed407aed46712a280e4b6b9a6f09c689e7766dfb42263"
Nov 03 20:15:05.359550 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:15:05.359358    5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp_odh-ai-gateway-infra(08e20158-6af2-4cec-ba28-74866c9a2731)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp" podUID="08e20158-6af2-4cec-ba28-74866c9a2731"
Nov 03 20:15:05.362182 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:05.362152    5591 generic.go:356] "Generic (PLEG): container finished" podID="9344a596-282c-4719-a323-5771291e1cdb" containerID="aabbea3e7b283052e0853ae40ac285a4cf115694081dab051f85def6f6b41071" exitCode=60
Nov 03 20:15:05.362719 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:05.362704    5591 scope.go:122] "RemoveContainer" containerID="aabbea3e7b283052e0853ae40ac285a4cf115694081dab051f85def6f6b41071"
Nov 03 20:15:05.363130 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:15:05.363111    5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj_odh-ai-gateway-infra(9344a596-282c-4719-a323-5771291e1cdb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj" podUID="9344a596-282c-4719-a323-5771291e1cdb"
Nov 03 20:15:05.369448 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:05.369391    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp" event={"ID":"08e20158-6af2-4cec-ba28-74866c9a2731","Type":"ContainerDied","Data":"011b2e26ccf39c9305fed407aed46712a280e4b6b9a6f09c689e7766dfb42263"}
Nov 03 20:15:05.369613 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:05.369476    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj" event={"ID":"9344a596-282c-4719-a323-5771291e1cdb","Type":"ContainerDied","Data":"aabbea3e7b283052e0853ae40ac285a4cf115694081dab051f85def6f6b41071"}
Nov 03 20:15:05.370890 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:05.370857    5591 generic.go:356] "Generic (PLEG): container finished" podID="59f8a1f2-c5c6-4d93-8477-60ddff9ea8df" containerID="1d1c598de67a941b4ccf9e289ad25c121b8bbfca09a27ba4baa565753ffbceb2" exitCode=60
Nov 03 20:15:05.370890 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:05.370903    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb" event={"ID":"59f8a1f2-c5c6-4d93-8477-60ddff9ea8df","Type":"ContainerDied","Data":"1d1c598de67a941b4ccf9e289ad25c121b8bbfca09a27ba4baa565753ffbceb2"}
Nov 03 20:15:05.371573 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:05.371547    5591 scope.go:122] "RemoveContainer" containerID="1d1c598de67a941b4ccf9e289ad25c121b8bbfca09a27ba4baa565753ffbceb2"
Nov 03 20:15:05.372405 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:15:05.372378    5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb_odh-ai-gateway-infra(59f8a1f2-c5c6-4d93-8477-60ddff9ea8df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb" podUID="59f8a1f2-c5c6-4d93-8477-60ddff9ea8df"
Nov 03 20:15:05.375420 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:05.375398    5591 scope.go:122] "RemoveContainer" containerID="4e7cd6ff438c58ad0a8d6b9bcbf34e9d21baac2acf0e609a7b88daf026dd6906"
Nov 03 20:15:05.425128 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:05.425058    5591 scope.go:122] "RemoveContainer" containerID="1be50c7102b5064bff329ea64309d40a75cc11389cfdb08811e4e6cde3fd7ed0"
Nov 03 20:15:05.455296 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:05.455197    5591 scope.go:122] "RemoveContainer" containerID="57392c03f4e12a2a185fda40c07eb9e917cc39065f1bd96fdaafe4d49f0919d7"
Nov 03 20:15:05.505128 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:05.504672    5591 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 03 20:15:05.505128 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:05.504729    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt" podUID="120358c4-5509-4cc4-8522-647ddbde2d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 03 20:15:06.436602 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:06.434325    5591 scope.go:122] "RemoveContainer" containerID="011b2e26ccf39c9305fed407aed46712a280e4b6b9a6f09c689e7766dfb42263"
Nov 03 20:15:06.436602 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:15:06.434858    5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp_odh-ai-gateway-infra(08e20158-6af2-4cec-ba28-74866c9a2731)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp" podUID="08e20158-6af2-4cec-ba28-74866c9a2731"
Nov 03 20:15:06.442152 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:06.439267    5591 scope.go:122] "RemoveContainer" containerID="aabbea3e7b283052e0853ae40ac285a4cf115694081dab051f85def6f6b41071"
Nov 03 20:15:06.442152 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:15:06.439748    5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj_odh-ai-gateway-infra(9344a596-282c-4719-a323-5771291e1cdb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj" podUID="9344a596-282c-4719-a323-5771291e1cdb"
Nov 03 20:15:06.444382 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:06.444356    5591 scope.go:122] "RemoveContainer" containerID="1d1c598de67a941b4ccf9e289ad25c121b8bbfca09a27ba4baa565753ffbceb2"
Nov 03 20:15:06.444753 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:15:06.444725    5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb_odh-ai-gateway-infra(59f8a1f2-c5c6-4d93-8477-60ddff9ea8df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb" podUID="59f8a1f2-c5c6-4d93-8477-60ddff9ea8df"
Nov 03 20:15:06.447622 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:06.447132    5591 scope.go:122] "RemoveContainer" containerID="8d60ecb989c2c45e3d2a2aaa398350364227630ce949ecbdb35d37f56113af36"
Nov 03 20:15:06.447622 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:15:06.447528    5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf_odh-ai-gateway-infra(4f1d45f5-4ee7-427b-a26e-e14568a5cf51)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf" podUID="4f1d45f5-4ee7-427b-a26e-e14568a5cf51"
Nov 03 20:15:06.501383 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:06.501345    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:15:06.503224 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:06.503187    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:15:06.504154 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:06.504127    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:15:07.122959 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:07.122930    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62"
Nov 03 20:15:10.158021 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:10.157982    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6"
Nov 03 20:15:15.342655 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:15.342566    5591 scope.go:122] "RemoveContainer" containerID="011b2e26ccf39c9305fed407aed46712a280e4b6b9a6f09c689e7766dfb42263"
Nov 03 20:15:15.343116 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:15.342976    5591 scope.go:122] "RemoveContainer" containerID="1d1c598de67a941b4ccf9e289ad25c121b8bbfca09a27ba4baa565753ffbceb2"
Nov 03 20:15:16.341925 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:16.341893    5591 scope.go:122] "RemoveContainer" containerID="8d60ecb989c2c45e3d2a2aaa398350364227630ce949ecbdb35d37f56113af36"
Nov 03 20:15:16.342117 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:16.342069    5591 scope.go:122] "RemoveContainer" containerID="aabbea3e7b283052e0853ae40ac285a4cf115694081dab051f85def6f6b41071"
Nov 03 20:15:16.537971 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:16.537938    5591 generic.go:356] "Generic (PLEG): container finished" podID="08e20158-6af2-4cec-ba28-74866c9a2731" containerID="50fb833e7fd2953e05acfe4e16ab6bb2f5514af36f5a773871f23242a2cae636" exitCode=60
Nov 03 20:15:16.538457 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:16.538007    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp" event={"ID":"08e20158-6af2-4cec-ba28-74866c9a2731","Type":"ContainerDied","Data":"50fb833e7fd2953e05acfe4e16ab6bb2f5514af36f5a773871f23242a2cae636"}
Nov 03 20:15:16.538457 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:16.538078    5591 scope.go:122] "RemoveContainer" containerID="011b2e26ccf39c9305fed407aed46712a280e4b6b9a6f09c689e7766dfb42263"
Nov 03 20:15:16.538621 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:16.538602    5591 scope.go:122] "RemoveContainer" containerID="50fb833e7fd2953e05acfe4e16ab6bb2f5514af36f5a773871f23242a2cae636"
Nov 03 20:15:16.539084 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:15:16.539019    5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp_odh-ai-gateway-infra(08e20158-6af2-4cec-ba28-74866c9a2731)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp" podUID="08e20158-6af2-4cec-ba28-74866c9a2731"
Nov 03 20:15:16.540299 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:16.540278    5591 generic.go:356] "Generic (PLEG): container finished" podID="59f8a1f2-c5c6-4d93-8477-60ddff9ea8df" containerID="5e68050365abc7445faef5bcbd12de7b3d786881b90161689683a96763b55d03" exitCode=60
Nov 03 20:15:16.540401 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:16.540350    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb" event={"ID":"59f8a1f2-c5c6-4d93-8477-60ddff9ea8df","Type":"ContainerDied","Data":"5e68050365abc7445faef5bcbd12de7b3d786881b90161689683a96763b55d03"}
Nov 03 20:15:16.540779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:16.540764    5591 scope.go:122] "RemoveContainer" containerID="5e68050365abc7445faef5bcbd12de7b3d786881b90161689683a96763b55d03"
Nov 03 20:15:16.541129 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:15:16.541111    5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb_odh-ai-gateway-infra(59f8a1f2-c5c6-4d93-8477-60ddff9ea8df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb" podUID="59f8a1f2-c5c6-4d93-8477-60ddff9ea8df"
Nov 03 20:15:16.564472 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:16.564442    5591 scope.go:122] "RemoveContainer" containerID="1d1c598de67a941b4ccf9e289ad25c121b8bbfca09a27ba4baa565753ffbceb2"
Nov 03 20:15:17.552641 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:17.552602    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf" event={"ID":"4f1d45f5-4ee7-427b-a26e-e14568a5cf51","Type":"ContainerStarted","Data":"d2f8432d44572e0a826ddfa9516da53ce3b9dc86df324948e572b47cd928d443"}
Nov 03 20:15:17.556943 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:17.556915    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj" event={"ID":"9344a596-282c-4719-a323-5771291e1cdb","Type":"ContainerStarted","Data":"d288bb0ab2e5faf2e2cfc56d260a03919a84affe1d3a78c4bf1a011ff1649b14"}
Nov 03 20:15:17.570785 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:17.570753    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp"]
Nov 03 20:15:17.571060 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:17.571017    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf" podStartSLOduration=17.57100465 podStartE2EDuration="17.57100465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:17.570842974 +0000 UTC m=+2266.343135995" watchObservedRunningTime="2026-11-03 20:15:17.57100465 +0000 UTC m=+2266.343297670"
Nov 03 20:15:17.682233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:17.680007    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb"]
Nov 03 20:15:17.995937 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:17.995908    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp"
Nov 03 20:15:18.000820 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.000794    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb"
Nov 03 20:15:18.065852 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.065774    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08e20158-6af2-4cec-ba28-74866c9a2731-kube-api-access-bfvc2\" (UniqueName: \"kubernetes.io/projected/08e20158-6af2-4cec-ba28-74866c9a2731-kube-api-access-bfvc2\") pod \"08e20158-6af2-4cec-ba28-74866c9a2731\" (UID: \"08e20158-6af2-4cec-ba28-74866c9a2731\") "
Nov 03 20:15:18.066008 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.065909    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59f8a1f2-c5c6-4d93-8477-60ddff9ea8df-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59f8a1f2-c5c6-4d93-8477-60ddff9ea8df-maas-api-service-ca\") pod \"59f8a1f2-c5c6-4d93-8477-60ddff9ea8df\" (UID: \"59f8a1f2-c5c6-4d93-8477-60ddff9ea8df\") "
Nov 03 20:15:18.066008 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.065958    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08e20158-6af2-4cec-ba28-74866c9a2731-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08e20158-6af2-4cec-ba28-74866c9a2731-maas-api-service-ca\") pod \"08e20158-6af2-4cec-ba28-74866c9a2731\" (UID: \"08e20158-6af2-4cec-ba28-74866c9a2731\") "
Nov 03 20:15:18.066008 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.065977    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59f8a1f2-c5c6-4d93-8477-60ddff9ea8df-kube-api-access-mfm7c\" (UniqueName: \"kubernetes.io/projected/59f8a1f2-c5c6-4d93-8477-60ddff9ea8df-kube-api-access-mfm7c\") pod \"59f8a1f2-c5c6-4d93-8477-60ddff9ea8df\" (UID: \"59f8a1f2-c5c6-4d93-8477-60ddff9ea8df\") "
Nov 03 20:15:18.066436 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.066405    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59f8a1f2-c5c6-4d93-8477-60ddff9ea8df-maas-api-service-ca" pod "59f8a1f2-c5c6-4d93-8477-60ddff9ea8df" (UID: "59f8a1f2-c5c6-4d93-8477-60ddff9ea8df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:18.066525 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.066396    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08e20158-6af2-4cec-ba28-74866c9a2731-maas-api-service-ca" pod "08e20158-6af2-4cec-ba28-74866c9a2731" (UID: "08e20158-6af2-4cec-ba28-74866c9a2731"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:18.066935 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.066897    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08e20158-6af2-4cec-ba28-74866c9a2731-maas-api-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:18.066935 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.066921    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59f8a1f2-c5c6-4d93-8477-60ddff9ea8df-maas-api-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:18.068426 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.068392    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08e20158-6af2-4cec-ba28-74866c9a2731-kube-api-access-bfvc2" pod "08e20158-6af2-4cec-ba28-74866c9a2731" (UID: "08e20158-6af2-4cec-ba28-74866c9a2731"). InnerVolumeSpecName "kube-api-access-bfvc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:18.068506 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.068444    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f8a1f2-c5c6-4d93-8477-60ddff9ea8df-kube-api-access-mfm7c" pod "59f8a1f2-c5c6-4d93-8477-60ddff9ea8df" (UID: "59f8a1f2-c5c6-4d93-8477-60ddff9ea8df"). InnerVolumeSpecName "kube-api-access-mfm7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:18.168370 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.168336    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfm7c\" (UniqueName: \"kubernetes.io/projected/59f8a1f2-c5c6-4d93-8477-60ddff9ea8df-kube-api-access-mfm7c\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:18.168370 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.168365    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfvc2\" (UniqueName: \"kubernetes.io/projected/08e20158-6af2-4cec-ba28-74866c9a2731-kube-api-access-bfvc2\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:18.570671 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.570630    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp" event={"ID":"08e20158-6af2-4cec-ba28-74866c9a2731","Type":"ContainerDied","Data":"9caee87bc4139271e94890e5a80df1d5b198df75aa4df6023c52a421768a6109"}
Nov 03 20:15:18.571187 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.570757    5591 scope.go:122] "RemoveContainer" containerID="50fb833e7fd2953e05acfe4e16ab6bb2f5514af36f5a773871f23242a2cae636"
Nov 03 20:15:18.571187 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.570884    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp"
Nov 03 20:15:18.574327 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.574299    5591 generic.go:356] "Generic (PLEG): container finished" podID="9344a596-282c-4719-a323-5771291e1cdb" containerID="d288bb0ab2e5faf2e2cfc56d260a03919a84affe1d3a78c4bf1a011ff1649b14" exitCode=60
Nov 03 20:15:18.574465 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.574427    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj" event={"ID":"9344a596-282c-4719-a323-5771291e1cdb","Type":"ContainerDied","Data":"d288bb0ab2e5faf2e2cfc56d260a03919a84affe1d3a78c4bf1a011ff1649b14"}
Nov 03 20:15:18.574934 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.574915    5591 scope.go:122] "RemoveContainer" containerID="d288bb0ab2e5faf2e2cfc56d260a03919a84affe1d3a78c4bf1a011ff1649b14"
Nov 03 20:15:18.575363 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:15:18.575333    5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj_odh-ai-gateway-infra(9344a596-282c-4719-a323-5771291e1cdb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj" podUID="9344a596-282c-4719-a323-5771291e1cdb"
Nov 03 20:15:18.577568 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.577542    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb" event={"ID":"59f8a1f2-c5c6-4d93-8477-60ddff9ea8df","Type":"ContainerDied","Data":"29faccaeef4a90e6f79089047ce9d93646650152f593e82cc4d35b2964f3acd1"}
Nov 03 20:15:18.577746 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.577730    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb"
Nov 03 20:15:18.585250 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.585180    5591 generic.go:356] "Generic (PLEG): container finished" podID="4f1d45f5-4ee7-427b-a26e-e14568a5cf51" containerID="d2f8432d44572e0a826ddfa9516da53ce3b9dc86df324948e572b47cd928d443" exitCode=60
Nov 03 20:15:18.585470 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.585409    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf" event={"ID":"4f1d45f5-4ee7-427b-a26e-e14568a5cf51","Type":"ContainerDied","Data":"d2f8432d44572e0a826ddfa9516da53ce3b9dc86df324948e572b47cd928d443"}
Nov 03 20:15:18.585825 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.585803    5591 scope.go:122] "RemoveContainer" containerID="d2f8432d44572e0a826ddfa9516da53ce3b9dc86df324948e572b47cd928d443"
Nov 03 20:15:18.587449 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:15:18.586160    5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf_odh-ai-gateway-infra(4f1d45f5-4ee7-427b-a26e-e14568a5cf51)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf" podUID="4f1d45f5-4ee7-427b-a26e-e14568a5cf51"
Nov 03 20:15:18.587449 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.587371    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf"]
Nov 03 20:15:18.596264 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.596246    5591 scope.go:122] "RemoveContainer" containerID="aabbea3e7b283052e0853ae40ac285a4cf115694081dab051f85def6f6b41071"
Nov 03 20:15:18.614720 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.614698    5591 scope.go:122] "RemoveContainer" containerID="5e68050365abc7445faef5bcbd12de7b3d786881b90161689683a96763b55d03"
Nov 03 20:15:18.625863 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.625834    5591 scope.go:122] "RemoveContainer" containerID="8d60ecb989c2c45e3d2a2aaa398350364227630ce949ecbdb35d37f56113af36"
Nov 03 20:15:18.632138 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.632115    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp"]
Nov 03 20:15:18.637053 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.637029    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2b5703-29895615-gmvdp"]
Nov 03 20:15:18.651178 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.651153    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj"]
Nov 03 20:15:18.670112 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.670074    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb"]
Nov 03 20:15:18.685813 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:18.685549    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-9f46e5-29895615-6n7pb"]
Nov 03 20:15:19.372792 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:19.372756    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08e20158-6af2-4cec-ba28-74866c9a2731" path="/var/lib/kubelet/pods/08e20158-6af2-4cec-ba28-74866c9a2731/volumes"
Nov 03 20:15:19.373838 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:19.373805    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59f8a1f2-c5c6-4d93-8477-60ddff9ea8df" path="/var/lib/kubelet/pods/59f8a1f2-c5c6-4d93-8477-60ddff9ea8df/volumes"
Nov 03 20:15:19.866877 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:19.866851    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj"
Nov 03 20:15:19.869982 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:19.869961    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf"
Nov 03 20:15:19.910693 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:19.910662    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9344a596-282c-4719-a323-5771291e1cdb-kube-api-access-b4fd5\" (UniqueName: \"kubernetes.io/projected/9344a596-282c-4719-a323-5771291e1cdb-kube-api-access-b4fd5\") pod \"9344a596-282c-4719-a323-5771291e1cdb\" (UID: \"9344a596-282c-4719-a323-5771291e1cdb\") "
Nov 03 20:15:19.910998 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:19.910781    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f1d45f5-4ee7-427b-a26e-e14568a5cf51-kube-api-access-4tdsz\" (UniqueName: \"kubernetes.io/projected/4f1d45f5-4ee7-427b-a26e-e14568a5cf51-kube-api-access-4tdsz\") pod \"4f1d45f5-4ee7-427b-a26e-e14568a5cf51\" (UID: \"4f1d45f5-4ee7-427b-a26e-e14568a5cf51\") "
Nov 03 20:15:19.911092 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:19.911024    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9344a596-282c-4719-a323-5771291e1cdb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9344a596-282c-4719-a323-5771291e1cdb-maas-api-service-ca\") pod \"9344a596-282c-4719-a323-5771291e1cdb\" (UID: \"9344a596-282c-4719-a323-5771291e1cdb\") "
Nov 03 20:15:19.911219 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:19.911190    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f1d45f5-4ee7-427b-a26e-e14568a5cf51-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f1d45f5-4ee7-427b-a26e-e14568a5cf51-maas-api-service-ca\") pod \"4f1d45f5-4ee7-427b-a26e-e14568a5cf51\" (UID: \"4f1d45f5-4ee7-427b-a26e-e14568a5cf51\") "
Nov 03 20:15:19.911810 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:19.911774    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9344a596-282c-4719-a323-5771291e1cdb-maas-api-service-ca" pod "9344a596-282c-4719-a323-5771291e1cdb" (UID: "9344a596-282c-4719-a323-5771291e1cdb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:19.911935 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:19.911847    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f1d45f5-4ee7-427b-a26e-e14568a5cf51-maas-api-service-ca" pod "4f1d45f5-4ee7-427b-a26e-e14568a5cf51" (UID: "4f1d45f5-4ee7-427b-a26e-e14568a5cf51"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:19.912923 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:19.912160    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f1d45f5-4ee7-427b-a26e-e14568a5cf51-maas-api-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:19.912923 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:19.912189    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9344a596-282c-4719-a323-5771291e1cdb-maas-api-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:19.914645 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:19.914613    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9344a596-282c-4719-a323-5771291e1cdb-kube-api-access-b4fd5" pod "9344a596-282c-4719-a323-5771291e1cdb" (UID: "9344a596-282c-4719-a323-5771291e1cdb"). InnerVolumeSpecName "kube-api-access-b4fd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:19.915085 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:19.915058    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f1d45f5-4ee7-427b-a26e-e14568a5cf51-kube-api-access-4tdsz" pod "4f1d45f5-4ee7-427b-a26e-e14568a5cf51" (UID: "4f1d45f5-4ee7-427b-a26e-e14568a5cf51"). InnerVolumeSpecName "kube-api-access-4tdsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:20.014360 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:20.014324    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tdsz\" (UniqueName: \"kubernetes.io/projected/4f1d45f5-4ee7-427b-a26e-e14568a5cf51-kube-api-access-4tdsz\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:20.014360 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:20.014356    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4fd5\" (UniqueName: \"kubernetes.io/projected/9344a596-282c-4719-a323-5771291e1cdb-kube-api-access-b4fd5\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:20.614605 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:20.614577    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj"
Nov 03 20:15:20.614815 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:20.614614    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj" event={"ID":"9344a596-282c-4719-a323-5771291e1cdb","Type":"ContainerDied","Data":"ae849fca72c83c2c6c8ccc7078daebeca9ccd9c759c03d4bf4cd06f48ab92421"}
Nov 03 20:15:20.614815 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:20.614688    5591 scope.go:122] "RemoveContainer" containerID="d288bb0ab2e5faf2e2cfc56d260a03919a84affe1d3a78c4bf1a011ff1649b14"
Nov 03 20:15:20.616927 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:20.616898    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf"
Nov 03 20:15:20.617073 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:20.616933    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf" event={"ID":"4f1d45f5-4ee7-427b-a26e-e14568a5cf51","Type":"ContainerDied","Data":"ecb2ff8b500b3be01dba68f716ad3422a2ebc9a3a13f599d8532a747a0ee1c4d"}
Nov 03 20:15:20.634887 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:20.634862    5591 scope.go:122] "RemoveContainer" containerID="d2f8432d44572e0a826ddfa9516da53ce3b9dc86df324948e572b47cd928d443"
Nov 03 20:15:20.652020 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:20.651989    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf"]
Nov 03 20:15:20.656734 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:20.656705    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d3d0df-29895615-5jjvf"]
Nov 03 20:15:20.670349 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:20.670315    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj"]
Nov 03 20:15:20.674977 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:20.674946    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e1546-29895615-5sfrj"]
Nov 03 20:15:21.353880 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:21.353840    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f1d45f5-4ee7-427b-a26e-e14568a5cf51" path="/var/lib/kubelet/pods/4f1d45f5-4ee7-427b-a26e-e14568a5cf51/volumes"
Nov 03 20:15:21.354667 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:21.354640    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9344a596-282c-4719-a323-5771291e1cdb" path="/var/lib/kubelet/pods/9344a596-282c-4719-a323-5771291e1cdb/volumes"
Nov 03 20:15:26.542895 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:26.542852    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:15:28.785353 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:15:28.785271    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:15:36.491178 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:36.491137    5591 scope.go:122] "RemoveContainer" containerID="197468be71b58f838fae5ea39a3cffe8d90a90619fc5175bcdbc1aec4180d143"
Nov 03 20:15:43.566318 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:43.558338    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62"]
Nov 03 20:15:44.446748 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:44.446711    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"]
Nov 03 20:15:44.446965 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:44.446942    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68" podUID="146cc739-235f-46e7-a767-1734c9127951" containerName="istio-proxy" containerID="cri-o://f1a723a7fa352d67f613b92b63e66f01243a440ad53dc6019a675f6d47638b5e" gracePeriod=30
Nov 03 20:15:48.559746 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:48.559699    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62" podUID="6e4fb2e7-9e81-4958-ac0e-6204fae57dc9" containerName="payload-processing" containerID="cri-o://02562140506cf26044cef48f452a820cb59207d1b1ae064f0d34e93fdb1fdd79" gracePeriod=25
Nov 03 20:15:48.841222 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:48.841172    5591 generic.go:356] "Generic (PLEG): container finished" podID="6e4fb2e7-9e81-4958-ac0e-6204fae57dc9" containerID="02562140506cf26044cef48f452a820cb59207d1b1ae064f0d34e93fdb1fdd79" exitCode=0
Nov 03 20:15:48.841361 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:48.841246    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62" event={"ID":"6e4fb2e7-9e81-4958-ac0e-6204fae57dc9","Type":"ContainerDied","Data":"02562140506cf26044cef48f452a820cb59207d1b1ae064f0d34e93fdb1fdd79"}
Nov 03 20:15:48.872557 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:48.872526    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62"
Nov 03 20:15:48.928645 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:48.928592    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-plugins-config-volume\") pod \"6e4fb2e7-9e81-4958-ac0e-6204fae57dc9\" (UID: \"6e4fb2e7-9e81-4958-ac0e-6204fae57dc9\") "
Nov 03 20:15:48.928863 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:48.928792    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-routing-overlay-volume\") pod \"6e4fb2e7-9e81-4958-ac0e-6204fae57dc9\" (UID: \"6e4fb2e7-9e81-4958-ac0e-6204fae57dc9\") "
Nov 03 20:15:48.928863 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:48.928837    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-provider-credentials\") pod \"6e4fb2e7-9e81-4958-ac0e-6204fae57dc9\" (UID: \"6e4fb2e7-9e81-4958-ac0e-6204fae57dc9\") "
Nov 03 20:15:48.928994 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:48.928981    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-kube-api-access-wfs7s\" (UniqueName: \"kubernetes.io/projected/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-kube-api-access-wfs7s\") pod \"6e4fb2e7-9e81-4958-ac0e-6204fae57dc9\" (UID: \"6e4fb2e7-9e81-4958-ac0e-6204fae57dc9\") "
Nov 03 20:15:48.929091 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:48.929061    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-plugins-config-volume" pod "6e4fb2e7-9e81-4958-ac0e-6204fae57dc9" (UID: "6e4fb2e7-9e81-4958-ac0e-6204fae57dc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:48.929791 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:48.929761    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-routing-overlay-volume" pod "6e4fb2e7-9e81-4958-ac0e-6204fae57dc9" (UID: "6e4fb2e7-9e81-4958-ac0e-6204fae57dc9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:48.930054 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:48.929985    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:48.930054 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:48.930019    5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-routing-overlay-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:48.931921 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:48.931893    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-kube-api-access-wfs7s" pod "6e4fb2e7-9e81-4958-ac0e-6204fae57dc9" (UID: "6e4fb2e7-9e81-4958-ac0e-6204fae57dc9"). InnerVolumeSpecName "kube-api-access-wfs7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:48.932025 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:48.931942    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-provider-credentials" pod "6e4fb2e7-9e81-4958-ac0e-6204fae57dc9" (UID: "6e4fb2e7-9e81-4958-ac0e-6204fae57dc9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:49.031816 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:49.031778    5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-provider-credentials\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:49.031816 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:49.031810    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfs7s\" (UniqueName: \"kubernetes.io/projected/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9-kube-api-access-wfs7s\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:49.859292 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:49.859255    5591 generic.go:356] "Generic (PLEG): container finished" podID="146cc739-235f-46e7-a767-1734c9127951" containerID="f1a723a7fa352d67f613b92b63e66f01243a440ad53dc6019a675f6d47638b5e" exitCode=0
Nov 03 20:15:49.859766 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:49.859748    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68" event={"ID":"146cc739-235f-46e7-a767-1734c9127951","Type":"ContainerDied","Data":"f1a723a7fa352d67f613b92b63e66f01243a440ad53dc6019a675f6d47638b5e"}
Nov 03 20:15:49.862190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:49.862172    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62"
Nov 03 20:15:49.862441 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:49.862322    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62" event={"ID":"6e4fb2e7-9e81-4958-ac0e-6204fae57dc9","Type":"ContainerDied","Data":"eec682e48b7051b22b09b1a76b9cb7052b9f6874bfd55d8c0dfca232fa3b43d4"}
Nov 03 20:15:49.862591 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:49.862576    5591 scope.go:122] "RemoveContainer" containerID="02562140506cf26044cef48f452a820cb59207d1b1ae064f0d34e93fdb1fdd79"
Nov 03 20:15:49.889488 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:49.889437    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62"]
Nov 03 20:15:49.893324 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:49.893293    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4c2a28/payload-processing-external-model-e2e-extmod-b-4c2a28-59d66db62"]
Nov 03 20:15:50.493239 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.488972    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:15:50.510251 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.510046    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-workload-certs\") pod \"146cc739-235f-46e7-a767-1734c9127951\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") "
Nov 03 20:15:50.510251 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.510151    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/146cc739-235f-46e7-a767-1734c9127951-istio-token\" (UniqueName: \"kubernetes.io/projected/146cc739-235f-46e7-a767-1734c9127951-istio-token\") pod \"146cc739-235f-46e7-a767-1734c9127951\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") "
Nov 03 20:15:50.510540 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.510377    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/146cc739-235f-46e7-a767-1734c9127951-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146cc739-235f-46e7-a767-1734c9127951-istio-podinfo\") pod \"146cc739-235f-46e7-a767-1734c9127951\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") "
Nov 03 20:15:50.510540 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.510388    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-workload-certs" pod "146cc739-235f-46e7-a767-1734c9127951" (UID: "146cc739-235f-46e7-a767-1734c9127951"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:15:50.510540 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.510437    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-istio-envoy\") pod \"146cc739-235f-46e7-a767-1734c9127951\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") "
Nov 03 20:15:50.510540 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.510476    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-istio-data\") pod \"146cc739-235f-46e7-a767-1734c9127951\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") "
Nov 03 20:15:50.510540 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.510538    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-credential-socket\") pod \"146cc739-235f-46e7-a767-1734c9127951\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") "
Nov 03 20:15:50.510793 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.510623    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/146cc739-235f-46e7-a767-1734c9127951-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146cc739-235f-46e7-a767-1734c9127951-istiod-ca-cert\") pod \"146cc739-235f-46e7-a767-1734c9127951\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") "
Nov 03 20:15:50.510793 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.510736    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-workload-socket\") pod \"146cc739-235f-46e7-a767-1734c9127951\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") "
Nov 03 20:15:50.510861 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.510808    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/146cc739-235f-46e7-a767-1734c9127951-kube-api-access-s6k2k\" (UniqueName: \"kubernetes.io/projected/146cc739-235f-46e7-a767-1734c9127951-kube-api-access-s6k2k\") pod \"146cc739-235f-46e7-a767-1734c9127951\" (UID: \"146cc739-235f-46e7-a767-1734c9127951\") "
Nov 03 20:15:50.514253 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.511402    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-istio-data" pod "146cc739-235f-46e7-a767-1734c9127951" (UID: "146cc739-235f-46e7-a767-1734c9127951"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:15:50.514253 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.511707    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:50.514253 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.511731    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:50.514253 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.511958    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146cc739-235f-46e7-a767-1734c9127951-istiod-ca-cert" pod "146cc739-235f-46e7-a767-1734c9127951" (UID: "146cc739-235f-46e7-a767-1734c9127951"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:15:50.514253 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.512224    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-credential-socket" pod "146cc739-235f-46e7-a767-1734c9127951" (UID: "146cc739-235f-46e7-a767-1734c9127951"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:15:50.515572 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.515550    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146cc739-235f-46e7-a767-1734c9127951-istio-token" pod "146cc739-235f-46e7-a767-1734c9127951" (UID: "146cc739-235f-46e7-a767-1734c9127951"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:50.521703 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.521668    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-istio-envoy" pod "146cc739-235f-46e7-a767-1734c9127951" (UID: "146cc739-235f-46e7-a767-1734c9127951"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:15:50.528077 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.528049    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-workload-socket" pod "146cc739-235f-46e7-a767-1734c9127951" (UID: "146cc739-235f-46e7-a767-1734c9127951"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:15:50.536315 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.533496    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/146cc739-235f-46e7-a767-1734c9127951-istio-podinfo" pod "146cc739-235f-46e7-a767-1734c9127951" (UID: "146cc739-235f-46e7-a767-1734c9127951"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:15:50.537167 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.537140    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146cc739-235f-46e7-a767-1734c9127951-kube-api-access-s6k2k" pod "146cc739-235f-46e7-a767-1734c9127951" (UID: "146cc739-235f-46e7-a767-1734c9127951"). InnerVolumeSpecName "kube-api-access-s6k2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:15:50.615353 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.615317    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146cc739-235f-46e7-a767-1734c9127951-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:50.615568 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.615361    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:50.615568 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.615378    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:50.615568 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.615394    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146cc739-235f-46e7-a767-1734c9127951-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:50.615568 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.615408    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146cc739-235f-46e7-a767-1734c9127951-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:50.615568 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.615423    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6k2k\" (UniqueName: \"kubernetes.io/projected/146cc739-235f-46e7-a767-1734c9127951-kube-api-access-s6k2k\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:50.615568 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.615438    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/146cc739-235f-46e7-a767-1734c9127951-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:15:50.873177 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.873143    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"
Nov 03 20:15:50.873177 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.873146    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68" event={"ID":"146cc739-235f-46e7-a767-1734c9127951","Type":"ContainerDied","Data":"9a0294b249a094e710039c55d8364ff1498793d8689b5cd2e33dfc4746d7ae8a"}
Nov 03 20:15:50.873667 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.873274    5591 scope.go:122] "RemoveContainer" containerID="f1a723a7fa352d67f613b92b63e66f01243a440ad53dc6019a675f6d47638b5e"
Nov 03 20:15:50.906296 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.906261    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"]
Nov 03 20:15:50.910647 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:50.910618    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f154ac-openshift-default-6f556bf4cc-6jz68"]
Nov 03 20:15:51.357691 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:51.357657    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="146cc739-235f-46e7-a767-1734c9127951" path="/var/lib/kubelet/pods/146cc739-235f-46e7-a767-1734c9127951/volumes"
Nov 03 20:15:51.358284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:51.358262    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e4fb2e7-9e81-4958-ac0e-6204fae57dc9" path="/var/lib/kubelet/pods/6e4fb2e7-9e81-4958-ac0e-6204fae57dc9/volumes"
Nov 03 20:15:52.678186 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.678112    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88"]
Nov 03 20:15:52.679093 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679070    5591 cpu_manager.go:414] "containerMap: removing container" podUID="9344a596-282c-4719-a323-5771291e1cdb" containerName="cleanup"
Nov 03 20:15:52.679093 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679091    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9344a596-282c-4719-a323-5771291e1cdb" containerName="cleanup"
Nov 03 20:15:52.679338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679107    5591 cpu_manager.go:414] "containerMap: removing container" podUID="08e20158-6af2-4cec-ba28-74866c9a2731" containerName="cleanup"
Nov 03 20:15:52.679338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679113    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08e20158-6af2-4cec-ba28-74866c9a2731" containerName="cleanup"
Nov 03 20:15:52.679338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679124    5591 cpu_manager.go:414] "containerMap: removing container" podUID="9344a596-282c-4719-a323-5771291e1cdb" containerName="cleanup"
Nov 03 20:15:52.679338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679134    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9344a596-282c-4719-a323-5771291e1cdb" containerName="cleanup"
Nov 03 20:15:52.679338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679153    5591 cpu_manager.go:414] "containerMap: removing container" podUID="08e20158-6af2-4cec-ba28-74866c9a2731" containerName="cleanup"
Nov 03 20:15:52.679338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679160    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08e20158-6af2-4cec-ba28-74866c9a2731" containerName="cleanup"
Nov 03 20:15:52.679338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679179    5591 cpu_manager.go:414] "containerMap: removing container" podUID="4f1d45f5-4ee7-427b-a26e-e14568a5cf51" containerName="cleanup"
Nov 03 20:15:52.679338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679186    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f1d45f5-4ee7-427b-a26e-e14568a5cf51" containerName="cleanup"
Nov 03 20:15:52.679338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679224    5591 cpu_manager.go:414] "containerMap: removing container" podUID="08e20158-6af2-4cec-ba28-74866c9a2731" containerName="cleanup"
Nov 03 20:15:52.679338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679231    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08e20158-6af2-4cec-ba28-74866c9a2731" containerName="cleanup"
Nov 03 20:15:52.679338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679268    5591 cpu_manager.go:414] "containerMap: removing container" podUID="59f8a1f2-c5c6-4d93-8477-60ddff9ea8df" containerName="cleanup"
Nov 03 20:15:52.679338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679274    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f8a1f2-c5c6-4d93-8477-60ddff9ea8df" containerName="cleanup"
Nov 03 20:15:52.679338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679291    5591 cpu_manager.go:414] "containerMap: removing container" podUID="4f1d45f5-4ee7-427b-a26e-e14568a5cf51" containerName="cleanup"
Nov 03 20:15:52.679338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679297    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f1d45f5-4ee7-427b-a26e-e14568a5cf51" containerName="cleanup"
Nov 03 20:15:52.679338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679320    5591 cpu_manager.go:414] "containerMap: removing container" podUID="146cc739-235f-46e7-a767-1734c9127951" containerName="istio-proxy"
Nov 03 20:15:52.679338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679326    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="146cc739-235f-46e7-a767-1734c9127951" containerName="istio-proxy"
Nov 03 20:15:52.679908 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679348    5591 cpu_manager.go:414] "containerMap: removing container" podUID="6e4fb2e7-9e81-4958-ac0e-6204fae57dc9" containerName="payload-processing"
Nov 03 20:15:52.679908 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679353    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e4fb2e7-9e81-4958-ac0e-6204fae57dc9" containerName="payload-processing"
Nov 03 20:15:52.679908 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679360    5591 cpu_manager.go:414] "containerMap: removing container" podUID="59f8a1f2-c5c6-4d93-8477-60ddff9ea8df" containerName="cleanup"
Nov 03 20:15:52.679908 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679364    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f8a1f2-c5c6-4d93-8477-60ddff9ea8df" containerName="cleanup"
Nov 03 20:15:52.679908 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679526    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="59f8a1f2-c5c6-4d93-8477-60ddff9ea8df" containerName="cleanup"
Nov 03 20:15:52.679908 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679539    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f1d45f5-4ee7-427b-a26e-e14568a5cf51" containerName="cleanup"
Nov 03 20:15:52.679908 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679544    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="9344a596-282c-4719-a323-5771291e1cdb" containerName="cleanup"
Nov 03 20:15:52.679908 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679551    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f1d45f5-4ee7-427b-a26e-e14568a5cf51" containerName="cleanup"
Nov 03 20:15:52.679908 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679555    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="146cc739-235f-46e7-a767-1734c9127951" containerName="istio-proxy"
Nov 03 20:15:52.679908 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679562    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="59f8a1f2-c5c6-4d93-8477-60ddff9ea8df" containerName="cleanup"
Nov 03 20:15:52.679908 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679569    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e4fb2e7-9e81-4958-ac0e-6204fae57dc9" containerName="payload-processing"
Nov 03 20:15:52.679908 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679574    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="59f8a1f2-c5c6-4d93-8477-60ddff9ea8df" containerName="cleanup"
Nov 03 20:15:52.679908 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679583    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="08e20158-6af2-4cec-ba28-74866c9a2731" containerName="cleanup"
Nov 03 20:15:52.679908 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679587    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="9344a596-282c-4719-a323-5771291e1cdb" containerName="cleanup"
Nov 03 20:15:52.679908 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679593    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="08e20158-6af2-4cec-ba28-74866c9a2731" containerName="cleanup"
Nov 03 20:15:52.679908 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679599    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f1d45f5-4ee7-427b-a26e-e14568a5cf51" containerName="cleanup"
Nov 03 20:15:52.679908 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.679605    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="08e20158-6af2-4cec-ba28-74866c9a2731" containerName="cleanup"
Nov 03 20:15:52.685448 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.685425    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88"
Nov 03 20:15:52.690904 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.690879    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-eb69cb\"/\"agc-cred-90b5ef16b936\""
Nov 03 20:15:52.692371 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.692349    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-eb69cb\"/\"kube-root-ca.crt\""
Nov 03 20:15:52.692524 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.692421    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-eb69cb\"/\"routing-overlay\""
Nov 03 20:15:52.692524 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.692461    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-eb69cb\"/\"openshift-service-ca.crt\""
Nov 03 20:15:52.692663 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.692549    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-eb69cb\"/\"payload-processing-external-model-plugins-e2e-extmod-a-eb69cb\""
Nov 03 20:15:52.697333 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.697072    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88"]
Nov 03 20:15:52.834275 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.834234    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88\" (UID: \"f639aa8a-ee24-45d2-a66c-1f363d7c6dca\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88"
Nov 03 20:15:52.834732 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.834444    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88\" (UID: \"f639aa8a-ee24-45d2-a66c-1f363d7c6dca\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88"
Nov 03 20:15:52.834732 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.834517    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88\" (UID: \"f639aa8a-ee24-45d2-a66c-1f363d7c6dca\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88"
Nov 03 20:15:52.834732 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.834725    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqqcf\" (UniqueName: \"kubernetes.io/projected/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-kube-api-access-vqqcf\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88\" (UID: \"f639aa8a-ee24-45d2-a66c-1f363d7c6dca\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88"
Nov 03 20:15:52.948300 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.947378    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqqcf\" (UniqueName: \"kubernetes.io/projected/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-kube-api-access-vqqcf\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88\" (UID: \"f639aa8a-ee24-45d2-a66c-1f363d7c6dca\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88"
Nov 03 20:15:52.948300 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.948044    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88\" (UID: \"f639aa8a-ee24-45d2-a66c-1f363d7c6dca\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88"
Nov 03 20:15:52.972231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.948933    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88\" (UID: \"f639aa8a-ee24-45d2-a66c-1f363d7c6dca\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88"
Nov 03 20:15:52.972231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.949029    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88\" (UID: \"f639aa8a-ee24-45d2-a66c-1f363d7c6dca\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88"
Nov 03 20:15:52.972231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.949956    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88\" (UID: \"f639aa8a-ee24-45d2-a66c-1f363d7c6dca\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88"
Nov 03 20:15:52.972231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.950757    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88\" (UID: \"f639aa8a-ee24-45d2-a66c-1f363d7c6dca\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88"
Nov 03 20:15:52.991388 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.985183    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqqcf\" (UniqueName: \"kubernetes.io/projected/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-kube-api-access-vqqcf\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88\" (UID: \"f639aa8a-ee24-45d2-a66c-1f363d7c6dca\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88"
Nov 03 20:15:53.007486 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:52.997862    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88\" (UID: \"f639aa8a-ee24-45d2-a66c-1f363d7c6dca\") " pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88"
Nov 03 20:15:53.020462 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:53.020176    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88"
Nov 03 20:15:53.475065 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:53.475039    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88"]
Nov 03 20:15:53.475761 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:15:53.475736    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf639aa8a_ee24_45d2_a66c_1f363d7c6dca.slice/crio-14d50809c2d0757aff53624826241cbf8a18c535605e4fb17df2c5c1ec891c34 WatchSource:0}: Error finding container 14d50809c2d0757aff53624826241cbf8a18c535605e4fb17df2c5c1ec891c34: Status 404 returned error can't find the container with id 14d50809c2d0757aff53624826241cbf8a18c535605e4fb17df2c5c1ec891c34
Nov 03 20:15:54.002925 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.002878    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88" event={"ID":"f639aa8a-ee24-45d2-a66c-1f363d7c6dca","Type":"ContainerStarted","Data":"26ddbfd0eb68d4adc4dec0755feab73ff6e2657c1ad287c53f609fd60d0a398c"}
Nov 03 20:15:54.003446 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.003025    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88"
Nov 03 20:15:54.003446 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.003051    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88" event={"ID":"f639aa8a-ee24-45d2-a66c-1f363d7c6dca","Type":"ContainerStarted","Data":"14d50809c2d0757aff53624826241cbf8a18c535605e4fb17df2c5c1ec891c34"}
Nov 03 20:15:54.025821 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.025763    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88" podStartSLOduration=2.025744111 podStartE2EDuration="2.025744111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:54.023405448 +0000 UTC m=+2302.795698471" watchObservedRunningTime="2026-11-03 20:15:54.025744111 +0000 UTC m=+2302.798037130"
Nov 03 20:15:54.107944 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.107907    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv"]
Nov 03 20:15:54.108985 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.108964    5591 cpu_manager.go:414] "containerMap: removing container" podUID="59f8a1f2-c5c6-4d93-8477-60ddff9ea8df" containerName="cleanup"
Nov 03 20:15:54.108985 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.108985    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f8a1f2-c5c6-4d93-8477-60ddff9ea8df" containerName="cleanup"
Nov 03 20:15:54.109265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.109000    5591 cpu_manager.go:414] "containerMap: removing container" podUID="4f1d45f5-4ee7-427b-a26e-e14568a5cf51" containerName="cleanup"
Nov 03 20:15:54.109265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.109005    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f1d45f5-4ee7-427b-a26e-e14568a5cf51" containerName="cleanup"
Nov 03 20:15:54.109265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.109041    5591 cpu_manager.go:414] "containerMap: removing container" podUID="9344a596-282c-4719-a323-5771291e1cdb" containerName="cleanup"
Nov 03 20:15:54.109265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.109046    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9344a596-282c-4719-a323-5771291e1cdb" containerName="cleanup"
Nov 03 20:15:54.109265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.109233    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="9344a596-282c-4719-a323-5771291e1cdb" containerName="cleanup"
Nov 03 20:15:54.113816 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.113767    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv"
Nov 03 20:15:54.124269 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.123595    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8b7351\""
Nov 03 20:15:54.124269 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.123688    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"openshift-service-ca.crt\""
Nov 03 20:15:54.124269 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.123699    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"kube-root-ca.crt\""
Nov 03 20:15:54.124269 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.123880    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"agc-cred-4fdc12f87588\""
Nov 03 20:15:54.124269 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.123949    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"agc-cred-a4e8a8a359a4\""
Nov 03 20:15:54.124269 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.124067    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"payload-processing-external-model-e2e--3d0fafa7-dockercfg-mk2qh\""
Nov 03 20:15:54.124269 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.124122    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"agc-cred-262850ebfa93\""
Nov 03 20:15:54.124866 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.124281    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"routing-overlay\""
Nov 03 20:15:54.125364 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.125109    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv"]
Nov 03 20:15:54.192872 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.192834    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv\" (UID: \"0da83501-fa04-4dea-9ea2-fe64aeb9f15f\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv"
Nov 03 20:15:54.193076 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.193055    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bx2w\" (UniqueName: \"kubernetes.io/projected/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-kube-api-access-7bx2w\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv\" (UID: \"0da83501-fa04-4dea-9ea2-fe64aeb9f15f\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv"
Nov 03 20:15:54.193238 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.193219    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv\" (UID: \"0da83501-fa04-4dea-9ea2-fe64aeb9f15f\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv"
Nov 03 20:15:54.193353 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.193337    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv\" (UID: \"0da83501-fa04-4dea-9ea2-fe64aeb9f15f\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv"
Nov 03 20:15:54.298229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.297442    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv\" (UID: \"0da83501-fa04-4dea-9ea2-fe64aeb9f15f\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv"
Nov 03 20:15:54.329241 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.313256    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv\" (UID: \"0da83501-fa04-4dea-9ea2-fe64aeb9f15f\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv"
Nov 03 20:15:54.329241 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.313616    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bx2w\" (UniqueName: \"kubernetes.io/projected/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-kube-api-access-7bx2w\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv\" (UID: \"0da83501-fa04-4dea-9ea2-fe64aeb9f15f\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv"
Nov 03 20:15:54.329241 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.313830    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv\" (UID: \"0da83501-fa04-4dea-9ea2-fe64aeb9f15f\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv"
Nov 03 20:15:54.329241 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.314955    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv\" (UID: \"0da83501-fa04-4dea-9ea2-fe64aeb9f15f\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv"
Nov 03 20:15:54.329241 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.315567    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv\" (UID: \"0da83501-fa04-4dea-9ea2-fe64aeb9f15f\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv"
Nov 03 20:15:54.329241 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.326991    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv\" (UID: \"0da83501-fa04-4dea-9ea2-fe64aeb9f15f\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv"
Nov 03 20:15:54.342283 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.342010    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bx2w\" (UniqueName: \"kubernetes.io/projected/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-kube-api-access-7bx2w\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv\" (UID: \"0da83501-fa04-4dea-9ea2-fe64aeb9f15f\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv"
Nov 03 20:15:54.436225 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.435307    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv"
Nov 03 20:15:54.943785 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.923029    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr"]
Nov 03 20:15:54.951547 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.951513    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr"]
Nov 03 20:15:54.951720 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.951662    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr"
Nov 03 20:15:54.964185 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:54.964134    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv"]
Nov 03 20:15:55.015617 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:55.015562    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv" event={"ID":"0da83501-fa04-4dea-9ea2-fe64aeb9f15f","Type":"ContainerStarted","Data":"c8b91ff92402f23f1f15c8d9cffa31ffc461005a0e564d7f2ca2c66fd07d1c02"}
Nov 03 20:15:55.087768 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:55.087724    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4xdl\" (UniqueName: \"kubernetes.io/projected/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-kube-api-access-h4xdl\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr\" (UID: \"924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr"
Nov 03 20:15:55.088114 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:55.087976    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr\" (UID: \"924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr"
Nov 03 20:15:55.088114 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:55.088031    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr\" (UID: \"924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr"
Nov 03 20:15:55.089378 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:55.088388    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr\" (UID: \"924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr"
Nov 03 20:15:55.190293 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:55.190182    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr\" (UID: \"924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr"
Nov 03 20:15:55.190293 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:55.190260    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr\" (UID: \"924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr"
Nov 03 20:15:55.190559 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:55.190443    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr\" (UID: \"924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr"
Nov 03 20:15:55.190559 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:55.190554    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4xdl\" (UniqueName: \"kubernetes.io/projected/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-kube-api-access-h4xdl\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr\" (UID: \"924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr"
Nov 03 20:15:55.190898 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:55.190868    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr\" (UID: \"924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr"
Nov 03 20:15:55.191070 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:55.191045    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr\" (UID: \"924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr"
Nov 03 20:15:55.193141 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:55.193112    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr\" (UID: \"924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr"
Nov 03 20:15:55.203895 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:55.203849    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4xdl\" (UniqueName: \"kubernetes.io/projected/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-kube-api-access-h4xdl\") pod \"payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr\" (UID: \"924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8\") " pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr"
Nov 03 20:15:55.279159 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:55.279114    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr"
Nov 03 20:15:55.628673 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:15:55.628624    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod924d5a25_2bf0_4236_a1b4_93c3a5ceb8b8.slice/crio-fd10f480d445df101b94ef3451d3f6c1a500d35082bcd27a600286cf3e69ad0a WatchSource:0}: Error finding container fd10f480d445df101b94ef3451d3f6c1a500d35082bcd27a600286cf3e69ad0a: Status 404 returned error can't find the container with id fd10f480d445df101b94ef3451d3f6c1a500d35082bcd27a600286cf3e69ad0a
Nov 03 20:15:55.628673 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:55.628659    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr"]
Nov 03 20:15:56.025719 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:56.025680    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv" event={"ID":"0da83501-fa04-4dea-9ea2-fe64aeb9f15f","Type":"ContainerStarted","Data":"b04f4a6b6f9f8af5a6efcf0c16926d3246ed385f83ad78b064ac12bc3326f2af"}
Nov 03 20:15:56.026179 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:56.025934    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv"
Nov 03 20:15:56.027599 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:56.027569    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr" event={"ID":"924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8","Type":"ContainerStarted","Data":"43c8b80f9926a0afbccc8604627a356149f7eed06fa3c9911d18420612d9ee8b"}
Nov 03 20:15:56.027739 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:56.027614    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr" event={"ID":"924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8","Type":"ContainerStarted","Data":"fd10f480d445df101b94ef3451d3f6c1a500d35082bcd27a600286cf3e69ad0a"}
Nov 03 20:15:56.027739 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:56.027646    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr"
Nov 03 20:15:56.049808 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:56.049744    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv" podStartSLOduration=2.049730177 podStartE2EDuration="2.049730177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:56.045868675 +0000 UTC m=+2304.818161695" watchObservedRunningTime="2026-11-03 20:15:56.049730177 +0000 UTC m=+2304.822023194"
Nov 03 20:15:56.068276 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:15:56.068222    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr" podStartSLOduration=2.068194072 podStartE2EDuration="2.068194072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:56.066132577 +0000 UTC m=+2304.838425596" watchObservedRunningTime="2026-11-03 20:15:56.068194072 +0000 UTC m=+2304.840487081"
Nov 03 20:15:58.726548 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:15:58.726506    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:16:05.020429 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:05.020386    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88"
Nov 03 20:16:06.038443 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:06.035782    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr"]
Nov 03 20:16:06.044698 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:06.043946    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr"
Nov 03 20:16:06.047355 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:06.047076    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6"]
Nov 03 20:16:07.060348 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:07.060317    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv"
Nov 03 20:16:11.038315 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.038270    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr" podUID="924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8" containerName="payload-processing" containerID="cri-o://43c8b80f9926a0afbccc8604627a356149f7eed06fa3c9911d18420612d9ee8b" gracePeriod=25
Nov 03 20:16:11.047799 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.047766    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6" podUID="3b177f2e-45af-4fd7-b033-13baeac28f49" containerName="payload-processing" containerID="cri-o://e30d5d05f86bded49d6ec609ebabbcb522c3657f1fa1679f8da060b09ce073c0" gracePeriod=25
Nov 03 20:16:11.223611 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.223576    5591 generic.go:356] "Generic (PLEG): container finished" podID="3b177f2e-45af-4fd7-b033-13baeac28f49" containerID="e30d5d05f86bded49d6ec609ebabbcb522c3657f1fa1679f8da060b09ce073c0" exitCode=0
Nov 03 20:16:11.223845 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.223825    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6" event={"ID":"3b177f2e-45af-4fd7-b033-13baeac28f49","Type":"ContainerDied","Data":"e30d5d05f86bded49d6ec609ebabbcb522c3657f1fa1679f8da060b09ce073c0"}
Nov 03 20:16:11.225751 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.225726    5591 generic.go:356] "Generic (PLEG): container finished" podID="924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8" containerID="43c8b80f9926a0afbccc8604627a356149f7eed06fa3c9911d18420612d9ee8b" exitCode=0
Nov 03 20:16:11.225912 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.225784    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr" event={"ID":"924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8","Type":"ContainerDied","Data":"43c8b80f9926a0afbccc8604627a356149f7eed06fa3c9911d18420612d9ee8b"}
Nov 03 20:16:11.442237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.437692    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr"
Nov 03 20:16:11.470445 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.470419    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6"
Nov 03 20:16:11.526436 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.526398    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b177f2e-45af-4fd7-b033-13baeac28f49-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3b177f2e-45af-4fd7-b033-13baeac28f49-provider-credentials\") pod \"3b177f2e-45af-4fd7-b033-13baeac28f49\" (UID: \"3b177f2e-45af-4fd7-b033-13baeac28f49\") "
Nov 03 20:16:11.526608 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.526585    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-plugins-config-volume\") pod \"924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8\" (UID: \"924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8\") "
Nov 03 20:16:11.526661 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.526634    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-routing-overlay-volume\") pod \"924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8\" (UID: \"924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8\") "
Nov 03 20:16:11.526719 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.526666    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b177f2e-45af-4fd7-b033-13baeac28f49-kube-api-access-fbqn6\" (UniqueName: \"kubernetes.io/projected/3b177f2e-45af-4fd7-b033-13baeac28f49-kube-api-access-fbqn6\") pod \"3b177f2e-45af-4fd7-b033-13baeac28f49\" (UID: \"3b177f2e-45af-4fd7-b033-13baeac28f49\") "
Nov 03 20:16:11.526780 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.526723    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b177f2e-45af-4fd7-b033-13baeac28f49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b177f2e-45af-4fd7-b033-13baeac28f49-plugins-config-volume\") pod \"3b177f2e-45af-4fd7-b033-13baeac28f49\" (UID: \"3b177f2e-45af-4fd7-b033-13baeac28f49\") "
Nov 03 20:16:11.526780 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.526769    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-provider-credentials\") pod \"924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8\" (UID: \"924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8\") "
Nov 03 20:16:11.526895 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.526813    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b177f2e-45af-4fd7-b033-13baeac28f49-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3b177f2e-45af-4fd7-b033-13baeac28f49-routing-overlay-volume\") pod \"3b177f2e-45af-4fd7-b033-13baeac28f49\" (UID: \"3b177f2e-45af-4fd7-b033-13baeac28f49\") "
Nov 03 20:16:11.527082 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.527056    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-kube-api-access-h4xdl\" (UniqueName: \"kubernetes.io/projected/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-kube-api-access-h4xdl\") pod \"924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8\" (UID: \"924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8\") "
Nov 03 20:16:11.527187 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.527078    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-routing-overlay-volume" pod "924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8" (UID: "924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:16:11.527187 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.527103    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-plugins-config-volume" pod "924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8" (UID: "924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:16:11.527344 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.527256    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b177f2e-45af-4fd7-b033-13baeac28f49-plugins-config-volume" pod "3b177f2e-45af-4fd7-b033-13baeac28f49" (UID: "3b177f2e-45af-4fd7-b033-13baeac28f49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:16:11.527626 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.527599    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b177f2e-45af-4fd7-b033-13baeac28f49-routing-overlay-volume" pod "3b177f2e-45af-4fd7-b033-13baeac28f49" (UID: "3b177f2e-45af-4fd7-b033-13baeac28f49"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:16:11.528150 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.527961    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:11.528150 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.527993    5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-routing-overlay-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:11.528150 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.528007    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b177f2e-45af-4fd7-b033-13baeac28f49-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:11.528150 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.528023    5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3b177f2e-45af-4fd7-b033-13baeac28f49-routing-overlay-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:11.530472 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.530431    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-kube-api-access-h4xdl" pod "924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8" (UID: "924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8"). InnerVolumeSpecName "kube-api-access-h4xdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:16:11.530582 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.530556    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b177f2e-45af-4fd7-b033-13baeac28f49-provider-credentials" pod "3b177f2e-45af-4fd7-b033-13baeac28f49" (UID: "3b177f2e-45af-4fd7-b033-13baeac28f49"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:16:11.530656 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.530639    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b177f2e-45af-4fd7-b033-13baeac28f49-kube-api-access-fbqn6" pod "3b177f2e-45af-4fd7-b033-13baeac28f49" (UID: "3b177f2e-45af-4fd7-b033-13baeac28f49"). InnerVolumeSpecName "kube-api-access-fbqn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:16:11.532109 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.532088    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-provider-credentials" pod "924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8" (UID: "924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:16:11.629982 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.629902    5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-provider-credentials\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:11.629982 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.629930    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4xdl\" (UniqueName: \"kubernetes.io/projected/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8-kube-api-access-h4xdl\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:11.629982 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.629941    5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3b177f2e-45af-4fd7-b033-13baeac28f49-provider-credentials\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:11.629982 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:11.629950    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbqn6\" (UniqueName: \"kubernetes.io/projected/3b177f2e-45af-4fd7-b033-13baeac28f49-kube-api-access-fbqn6\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:12.236758 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:12.236720    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6"
Nov 03 20:16:12.236758 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:12.236735    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6" event={"ID":"3b177f2e-45af-4fd7-b033-13baeac28f49","Type":"ContainerDied","Data":"3a1fcc04ee54e2e74e89ec076bb674e166bf6da36cf3942e25112cb90a1c3db1"}
Nov 03 20:16:12.237358 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:12.236809    5591 scope.go:122] "RemoveContainer" containerID="e30d5d05f86bded49d6ec609ebabbcb522c3657f1fa1679f8da060b09ce073c0"
Nov 03 20:16:12.238785 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:12.238754    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr"
Nov 03 20:16:12.238909 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:12.238799    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr" event={"ID":"924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8","Type":"ContainerDied","Data":"fd10f480d445df101b94ef3451d3f6c1a500d35082bcd27a600286cf3e69ad0a"}
Nov 03 20:16:12.250696 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:12.250578    5591 scope.go:122] "RemoveContainer" containerID="43c8b80f9926a0afbccc8604627a356149f7eed06fa3c9911d18420612d9ee8b"
Nov 03 20:16:12.274989 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:12.274954    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6"]
Nov 03 20:16:12.284938 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:12.284907    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-8658vkfj6"]
Nov 03 20:16:12.291969 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:12.291932    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr"]
Nov 03 20:16:12.296872 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:12.296845    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-98529d/payload-processing-external-model-e2e-extmod-a-98529d-c78bnqrrr"]
Nov 03 20:16:13.353668 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:13.353638    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b177f2e-45af-4fd7-b033-13baeac28f49" path="/var/lib/kubelet/pods/3b177f2e-45af-4fd7-b033-13baeac28f49/volumes"
Nov 03 20:16:13.354058 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:13.354034    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8" path="/var/lib/kubelet/pods/924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8/volumes"
Nov 03 20:16:13.718328 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:13.714690    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:16:23.997939 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:23.997907    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf"]
Nov 03 20:16:23.998933 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:23.998915    5591 cpu_manager.go:414] "containerMap: removing container" podUID="3b177f2e-45af-4fd7-b033-13baeac28f49" containerName="payload-processing"
Nov 03 20:16:23.998995 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:23.998935    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b177f2e-45af-4fd7-b033-13baeac28f49" containerName="payload-processing"
Nov 03 20:16:23.998995 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:23.998958    5591 cpu_manager.go:414] "containerMap: removing container" podUID="924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8" containerName="payload-processing"
Nov 03 20:16:23.998995 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:23.998963    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8" containerName="payload-processing"
Nov 03 20:16:23.999112 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:23.999089    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b177f2e-45af-4fd7-b033-13baeac28f49" containerName="payload-processing"
Nov 03 20:16:23.999112 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:23.999102    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="924d5a25-2bf0-4236-a1b4-93c3a5ceb8b8" containerName="payload-processing"
Nov 03 20:16:24.004697 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:24.004673    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf"
Nov 03 20:16:24.010227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:24.008584    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-001b5c9f6a6a\""
Nov 03 20:16:24.010227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:24.009484    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 20:16:24.010227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:24.009883    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 20:16:24.010227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:24.009935    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 20:16:24.010537 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:24.010429    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-16659cd0d969\""
Nov 03 20:16:24.011486 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:24.011302    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-5lpr4\""
Nov 03 20:16:24.014691 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:24.012137    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 20:16:24.018560 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:24.018533    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf"]
Nov 03 20:16:24.168826 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:24.168783    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qzpb\" (UniqueName: \"kubernetes.io/projected/34043cb5-5fdd-4885-9403-5b87877de6cd-kube-api-access-4qzpb\") pod \"payload-processing-external-model-56bdcb5597-tcwmf\" (UID: \"34043cb5-5fdd-4885-9403-5b87877de6cd\") " pod="models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf"
Nov 03 20:16:24.169016 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:24.168965    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34043cb5-5fdd-4885-9403-5b87877de6cd-provider-credentials\") pod \"payload-processing-external-model-56bdcb5597-tcwmf\" (UID: \"34043cb5-5fdd-4885-9403-5b87877de6cd\") " pod="models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf"
Nov 03 20:16:24.169079 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:24.169016    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34043cb5-5fdd-4885-9403-5b87877de6cd-routing-overlay-volume\") pod \"payload-processing-external-model-56bdcb5597-tcwmf\" (UID: \"34043cb5-5fdd-4885-9403-5b87877de6cd\") " pod="models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf"
Nov 03 20:16:24.169137 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:24.169120    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34043cb5-5fdd-4885-9403-5b87877de6cd-plugins-config-volume\") pod \"payload-processing-external-model-56bdcb5597-tcwmf\" (UID: \"34043cb5-5fdd-4885-9403-5b87877de6cd\") " pod="models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf"
Nov 03 20:16:24.271414 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:24.271384    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qzpb\" (UniqueName: \"kubernetes.io/projected/34043cb5-5fdd-4885-9403-5b87877de6cd-kube-api-access-4qzpb\") pod \"payload-processing-external-model-56bdcb5597-tcwmf\" (UID: \"34043cb5-5fdd-4885-9403-5b87877de6cd\") " pod="models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf"
Nov 03 20:16:24.271636 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:24.271493    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34043cb5-5fdd-4885-9403-5b87877de6cd-provider-credentials\") pod \"payload-processing-external-model-56bdcb5597-tcwmf\" (UID: \"34043cb5-5fdd-4885-9403-5b87877de6cd\") " pod="models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf"
Nov 03 20:16:24.271636 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:24.271532    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34043cb5-5fdd-4885-9403-5b87877de6cd-routing-overlay-volume\") pod \"payload-processing-external-model-56bdcb5597-tcwmf\" (UID: \"34043cb5-5fdd-4885-9403-5b87877de6cd\") " pod="models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf"
Nov 03 20:16:24.271636 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:24.271599    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34043cb5-5fdd-4885-9403-5b87877de6cd-plugins-config-volume\") pod \"payload-processing-external-model-56bdcb5597-tcwmf\" (UID: \"34043cb5-5fdd-4885-9403-5b87877de6cd\") " pod="models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf"
Nov 03 20:16:24.272511 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:24.272481    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34043cb5-5fdd-4885-9403-5b87877de6cd-routing-overlay-volume\") pod \"payload-processing-external-model-56bdcb5597-tcwmf\" (UID: \"34043cb5-5fdd-4885-9403-5b87877de6cd\") " pod="models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf"
Nov 03 20:16:24.272777 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:24.272558    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34043cb5-5fdd-4885-9403-5b87877de6cd-plugins-config-volume\") pod \"payload-processing-external-model-56bdcb5597-tcwmf\" (UID: \"34043cb5-5fdd-4885-9403-5b87877de6cd\") " pod="models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf"
Nov 03 20:16:24.274586 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:24.274564    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34043cb5-5fdd-4885-9403-5b87877de6cd-provider-credentials\") pod \"payload-processing-external-model-56bdcb5597-tcwmf\" (UID: \"34043cb5-5fdd-4885-9403-5b87877de6cd\") " pod="models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf"
Nov 03 20:16:24.283708 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:24.283683    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qzpb\" (UniqueName: \"kubernetes.io/projected/34043cb5-5fdd-4885-9403-5b87877de6cd-kube-api-access-4qzpb\") pod \"payload-processing-external-model-56bdcb5597-tcwmf\" (UID: \"34043cb5-5fdd-4885-9403-5b87877de6cd\") " pod="models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf"
Nov 03 20:16:24.350294 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:24.350252    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf"
Nov 03 20:16:24.639842 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:16:24.639811    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34043cb5_5fdd_4885_9403_5b87877de6cd.slice/crio-1b5162c1f4df1cac8e6b625350f80307a297f598cd0b60e145b0e2b80f6c1b3f WatchSource:0}: Error finding container 1b5162c1f4df1cac8e6b625350f80307a297f598cd0b60e145b0e2b80f6c1b3f: Status 404 returned error can't find the container with id 1b5162c1f4df1cac8e6b625350f80307a297f598cd0b60e145b0e2b80f6c1b3f
Nov 03 20:16:24.640319 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:24.640289    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf"]
Nov 03 20:16:25.524991 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:25.524952    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf" event={"ID":"34043cb5-5fdd-4885-9403-5b87877de6cd","Type":"ContainerStarted","Data":"578b912228da7267dbec002a401cd1cb9c8386b5f58041c1d324ac29f0ac3bf9"}
Nov 03 20:16:25.525465 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:25.525011    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf" event={"ID":"34043cb5-5fdd-4885-9403-5b87877de6cd","Type":"ContainerStarted","Data":"1b5162c1f4df1cac8e6b625350f80307a297f598cd0b60e145b0e2b80f6c1b3f"}
Nov 03 20:16:25.525465 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:25.525049    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf"
Nov 03 20:16:25.547351 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:25.547291    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf" podStartSLOduration=2.5472726039999998 podStartE2EDuration="2.547272604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:25.543306349 +0000 UTC m=+2334.315599365" watchObservedRunningTime="2026-11-03 20:16:25.547272604 +0000 UTC m=+2334.319565615"
Nov 03 20:16:28.726726 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:16:28.726688    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:16:32.283453 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:32.283414    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg"]
Nov 03 20:16:32.290927 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:32.290071    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg"
Nov 03 20:16:32.298045 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:32.298014    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg"]
Nov 03 20:16:32.387101 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:32.387068    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c94dcae3-f45e-4e86-a0a7-b22bef29056e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg\" (UID: \"c94dcae3-f45e-4e86-a0a7-b22bef29056e\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg"
Nov 03 20:16:32.387431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:32.387408    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zprx8\" (UniqueName: \"kubernetes.io/projected/c94dcae3-f45e-4e86-a0a7-b22bef29056e-kube-api-access-zprx8\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg\" (UID: \"c94dcae3-f45e-4e86-a0a7-b22bef29056e\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg"
Nov 03 20:16:32.387547 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:32.387530    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c94dcae3-f45e-4e86-a0a7-b22bef29056e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg\" (UID: \"c94dcae3-f45e-4e86-a0a7-b22bef29056e\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg"
Nov 03 20:16:32.387597 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:32.387584    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c94dcae3-f45e-4e86-a0a7-b22bef29056e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg\" (UID: \"c94dcae3-f45e-4e86-a0a7-b22bef29056e\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg"
Nov 03 20:16:32.489172 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:32.489132    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c94dcae3-f45e-4e86-a0a7-b22bef29056e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg\" (UID: \"c94dcae3-f45e-4e86-a0a7-b22bef29056e\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg"
Nov 03 20:16:32.489399 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:32.489290    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zprx8\" (UniqueName: \"kubernetes.io/projected/c94dcae3-f45e-4e86-a0a7-b22bef29056e-kube-api-access-zprx8\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg\" (UID: \"c94dcae3-f45e-4e86-a0a7-b22bef29056e\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg"
Nov 03 20:16:32.489399 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:32.489348    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c94dcae3-f45e-4e86-a0a7-b22bef29056e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg\" (UID: \"c94dcae3-f45e-4e86-a0a7-b22bef29056e\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg"
Nov 03 20:16:32.489399 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:32.489379    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c94dcae3-f45e-4e86-a0a7-b22bef29056e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg\" (UID: \"c94dcae3-f45e-4e86-a0a7-b22bef29056e\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg"
Nov 03 20:16:32.490123 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:32.490096    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c94dcae3-f45e-4e86-a0a7-b22bef29056e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg\" (UID: \"c94dcae3-f45e-4e86-a0a7-b22bef29056e\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg"
Nov 03 20:16:32.490123 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:32.490112    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c94dcae3-f45e-4e86-a0a7-b22bef29056e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg\" (UID: \"c94dcae3-f45e-4e86-a0a7-b22bef29056e\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg"
Nov 03 20:16:32.492244 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:32.492194    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c94dcae3-f45e-4e86-a0a7-b22bef29056e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg\" (UID: \"c94dcae3-f45e-4e86-a0a7-b22bef29056e\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg"
Nov 03 20:16:32.502254 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:32.502230    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zprx8\" (UniqueName: \"kubernetes.io/projected/c94dcae3-f45e-4e86-a0a7-b22bef29056e-kube-api-access-zprx8\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg\" (UID: \"c94dcae3-f45e-4e86-a0a7-b22bef29056e\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg"
Nov 03 20:16:32.662098 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:32.662008    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg"
Nov 03 20:16:33.133234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:33.133177    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg"]
Nov 03 20:16:33.133459 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:16:33.133437    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc94dcae3_f45e_4e86_a0a7_b22bef29056e.slice/crio-2eada6776e90ffed23988b09ffc9c340c17e9d971069cda1a9783d85d776d234 WatchSource:0}: Error finding container 2eada6776e90ffed23988b09ffc9c340c17e9d971069cda1a9783d85d776d234: Status 404 returned error can't find the container with id 2eada6776e90ffed23988b09ffc9c340c17e9d971069cda1a9783d85d776d234
Nov 03 20:16:33.805521 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:33.805480    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg" event={"ID":"c94dcae3-f45e-4e86-a0a7-b22bef29056e","Type":"ContainerStarted","Data":"dd3ebf391e3fbd568def81b19d245298f07f38d0869d7b080fbecade5aa4aae3"}
Nov 03 20:16:33.805955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:33.805548    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg" event={"ID":"c94dcae3-f45e-4e86-a0a7-b22bef29056e","Type":"ContainerStarted","Data":"2eada6776e90ffed23988b09ffc9c340c17e9d971069cda1a9783d85d776d234"}
Nov 03 20:16:33.805955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:33.805571    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg"
Nov 03 20:16:33.829644 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:33.829597    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg" podStartSLOduration=1.829582829 podStartE2EDuration="1.829582829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:33.828307204 +0000 UTC m=+2342.600600222" watchObservedRunningTime="2026-11-03 20:16:33.829582829 +0000 UTC m=+2342.601875846"
Nov 03 20:16:36.559594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:36.559563    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf"
Nov 03 20:16:36.578496 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:36.578469    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf"]
Nov 03 20:16:36.740165 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:16:36.735358    5591 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-001b5c9f6a6a: secret "agc-cred-001b5c9f6a6a" not found
Nov 03 20:16:36.740165 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:16:36.735398    5591 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf: secret "agc-cred-001b5c9f6a6a" not found
Nov 03 20:16:36.740165 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:16:36.735489    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34043cb5-5fdd-4885-9403-5b87877de6cd-provider-credentials podName:34043cb5-5fdd-4885-9403-5b87877de6cd nodeName:}" failed. No retries permitted until 2026-11-03 20:16:37.235466838 +0000 UTC m=+2346.007759849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/34043cb5-5fdd-4885-9403-5b87877de6cd-provider-credentials") pod "payload-processing-external-model-56bdcb5597-tcwmf" (UID: "34043cb5-5fdd-4885-9403-5b87877de6cd") : secret "agc-cred-001b5c9f6a6a" not found
Nov 03 20:16:37.238444 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:16:37.238414    5591 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-001b5c9f6a6a: secret "agc-cred-001b5c9f6a6a" not found
Nov 03 20:16:37.238444 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:16:37.238444    5591 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf: secret "agc-cred-001b5c9f6a6a" not found
Nov 03 20:16:37.238685 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:16:37.238497    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34043cb5-5fdd-4885-9403-5b87877de6cd-provider-credentials podName:34043cb5-5fdd-4885-9403-5b87877de6cd nodeName:}" failed. No retries permitted until 2026-11-03 20:16:38.238483301 +0000 UTC m=+2347.010776310 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/34043cb5-5fdd-4885-9403-5b87877de6cd-provider-credentials") pod "payload-processing-external-model-56bdcb5597-tcwmf" (UID: "34043cb5-5fdd-4885-9403-5b87877de6cd") : secret "agc-cred-001b5c9f6a6a" not found
Nov 03 20:16:38.291484 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:16:38.291446    5591 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-001b5c9f6a6a: secret "agc-cred-001b5c9f6a6a" not found
Nov 03 20:16:38.291484 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:16:38.291484    5591 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf: secret "agc-cred-001b5c9f6a6a" not found
Nov 03 20:16:38.291943 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:16:38.291555    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34043cb5-5fdd-4885-9403-5b87877de6cd-provider-credentials podName:34043cb5-5fdd-4885-9403-5b87877de6cd nodeName:}" failed. No retries permitted until 2026-11-03 20:16:40.291531998 +0000 UTC m=+2349.063824994 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/34043cb5-5fdd-4885-9403-5b87877de6cd-provider-credentials") pod "payload-processing-external-model-56bdcb5597-tcwmf" (UID: "34043cb5-5fdd-4885-9403-5b87877de6cd") : secret "agc-cred-001b5c9f6a6a" not found
Nov 03 20:16:40.382768 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:16:40.382537    5591 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-001b5c9f6a6a: secret "agc-cred-001b5c9f6a6a" not found
Nov 03 20:16:40.382768 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:16:40.382574    5591 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf: secret "agc-cred-001b5c9f6a6a" not found
Nov 03 20:16:40.382768 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:16:40.382640    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34043cb5-5fdd-4885-9403-5b87877de6cd-provider-credentials podName:34043cb5-5fdd-4885-9403-5b87877de6cd nodeName:}" failed. No retries permitted until 2026-11-03 20:16:44.382619205 +0000 UTC m=+2353.154912223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/34043cb5-5fdd-4885-9403-5b87877de6cd-provider-credentials") pod "payload-processing-external-model-56bdcb5597-tcwmf" (UID: "34043cb5-5fdd-4885-9403-5b87877de6cd") : secret "agc-cred-001b5c9f6a6a" not found
Nov 03 20:16:41.847070 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:41.846961    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf" podUID="34043cb5-5fdd-4885-9403-5b87877de6cd" containerName="payload-processing" containerID="cri-o://578b912228da7267dbec002a401cd1cb9c8386b5f58041c1d324ac29f0ac3bf9" gracePeriod=25
Nov 03 20:16:42.718171 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.718141    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf"
Nov 03 20:16:42.727105 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.727080    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34043cb5-5fdd-4885-9403-5b87877de6cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34043cb5-5fdd-4885-9403-5b87877de6cd-plugins-config-volume\") pod \"34043cb5-5fdd-4885-9403-5b87877de6cd\" (UID: \"34043cb5-5fdd-4885-9403-5b87877de6cd\") "
Nov 03 20:16:42.727327 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.727185    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34043cb5-5fdd-4885-9403-5b87877de6cd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/34043cb5-5fdd-4885-9403-5b87877de6cd-provider-credentials\") pod \"34043cb5-5fdd-4885-9403-5b87877de6cd\" (UID: \"34043cb5-5fdd-4885-9403-5b87877de6cd\") "
Nov 03 20:16:42.727327 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.727264    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34043cb5-5fdd-4885-9403-5b87877de6cd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34043cb5-5fdd-4885-9403-5b87877de6cd-routing-overlay-volume\") pod \"34043cb5-5fdd-4885-9403-5b87877de6cd\" (UID: \"34043cb5-5fdd-4885-9403-5b87877de6cd\") "
Nov 03 20:16:42.727327 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.727286    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34043cb5-5fdd-4885-9403-5b87877de6cd-kube-api-access-4qzpb\" (UniqueName: \"kubernetes.io/projected/34043cb5-5fdd-4885-9403-5b87877de6cd-kube-api-access-4qzpb\") pod \"34043cb5-5fdd-4885-9403-5b87877de6cd\" (UID: \"34043cb5-5fdd-4885-9403-5b87877de6cd\") "
Nov 03 20:16:42.727590 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.727505    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34043cb5-5fdd-4885-9403-5b87877de6cd-plugins-config-volume" pod "34043cb5-5fdd-4885-9403-5b87877de6cd" (UID: "34043cb5-5fdd-4885-9403-5b87877de6cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:16:42.727659 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.727623    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34043cb5-5fdd-4885-9403-5b87877de6cd-routing-overlay-volume" pod "34043cb5-5fdd-4885-9403-5b87877de6cd" (UID: "34043cb5-5fdd-4885-9403-5b87877de6cd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:16:42.727993 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.727964    5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34043cb5-5fdd-4885-9403-5b87877de6cd-routing-overlay-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:42.727993 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.727987    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34043cb5-5fdd-4885-9403-5b87877de6cd-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:42.729780 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.729753    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34043cb5-5fdd-4885-9403-5b87877de6cd-kube-api-access-4qzpb" pod "34043cb5-5fdd-4885-9403-5b87877de6cd" (UID: "34043cb5-5fdd-4885-9403-5b87877de6cd"). InnerVolumeSpecName "kube-api-access-4qzpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:16:42.729896 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.729790    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34043cb5-5fdd-4885-9403-5b87877de6cd-provider-credentials" pod "34043cb5-5fdd-4885-9403-5b87877de6cd" (UID: "34043cb5-5fdd-4885-9403-5b87877de6cd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:16:42.766133 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.766096    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-65759456-22sfn"]
Nov 03 20:16:42.768015 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.767986    5591 cpu_manager.go:414] "containerMap: removing container" podUID="34043cb5-5fdd-4885-9403-5b87877de6cd" containerName="payload-processing"
Nov 03 20:16:42.768015 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.768016    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34043cb5-5fdd-4885-9403-5b87877de6cd" containerName="payload-processing"
Nov 03 20:16:42.768349 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.768331    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="34043cb5-5fdd-4885-9403-5b87877de6cd" containerName="payload-processing"
Nov 03 20:16:42.782995 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.782965    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-65759456-22sfn"]
Nov 03 20:16:42.783169 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.783084    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65759456-22sfn"
Nov 03 20:16:42.829341 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.829300    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/608043b9-c959-4e55-b909-f810cb6e884a-plugins-config-volume\") pod \"payload-processing-external-model-65759456-22sfn\" (UID: \"608043b9-c959-4e55-b909-f810cb6e884a\") " pod="models-as-a-service/payload-processing-external-model-65759456-22sfn"
Nov 03 20:16:42.829546 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.829464    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lkgw\" (UniqueName: \"kubernetes.io/projected/608043b9-c959-4e55-b909-f810cb6e884a-kube-api-access-7lkgw\") pod \"payload-processing-external-model-65759456-22sfn\" (UID: \"608043b9-c959-4e55-b909-f810cb6e884a\") " pod="models-as-a-service/payload-processing-external-model-65759456-22sfn"
Nov 03 20:16:42.829666 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.829647    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/608043b9-c959-4e55-b909-f810cb6e884a-provider-credentials\") pod \"payload-processing-external-model-65759456-22sfn\" (UID: \"608043b9-c959-4e55-b909-f810cb6e884a\") " pod="models-as-a-service/payload-processing-external-model-65759456-22sfn"
Nov 03 20:16:42.829751 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.829728    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/608043b9-c959-4e55-b909-f810cb6e884a-routing-overlay-volume\") pod \"payload-processing-external-model-65759456-22sfn\" (UID: \"608043b9-c959-4e55-b909-f810cb6e884a\") " pod="models-as-a-service/payload-processing-external-model-65759456-22sfn"
Nov 03 20:16:42.829976 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.829946    5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34043cb5-5fdd-4885-9403-5b87877de6cd-provider-credentials\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:42.830114 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.829980    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qzpb\" (UniqueName: \"kubernetes.io/projected/34043cb5-5fdd-4885-9403-5b87877de6cd-kube-api-access-4qzpb\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:42.987294 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.979996    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/608043b9-c959-4e55-b909-f810cb6e884a-provider-credentials\") pod \"payload-processing-external-model-65759456-22sfn\" (UID: \"608043b9-c959-4e55-b909-f810cb6e884a\") " pod="models-as-a-service/payload-processing-external-model-65759456-22sfn"
Nov 03 20:16:42.987294 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.980140    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/608043b9-c959-4e55-b909-f810cb6e884a-routing-overlay-volume\") pod \"payload-processing-external-model-65759456-22sfn\" (UID: \"608043b9-c959-4e55-b909-f810cb6e884a\") " pod="models-as-a-service/payload-processing-external-model-65759456-22sfn"
Nov 03 20:16:42.987294 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.980286    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/608043b9-c959-4e55-b909-f810cb6e884a-plugins-config-volume\") pod \"payload-processing-external-model-65759456-22sfn\" (UID: \"608043b9-c959-4e55-b909-f810cb6e884a\") " pod="models-as-a-service/payload-processing-external-model-65759456-22sfn"
Nov 03 20:16:42.987294 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.980405    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lkgw\" (UniqueName: \"kubernetes.io/projected/608043b9-c959-4e55-b909-f810cb6e884a-kube-api-access-7lkgw\") pod \"payload-processing-external-model-65759456-22sfn\" (UID: \"608043b9-c959-4e55-b909-f810cb6e884a\") " pod="models-as-a-service/payload-processing-external-model-65759456-22sfn"
Nov 03 20:16:42.987294 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.984266    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/608043b9-c959-4e55-b909-f810cb6e884a-routing-overlay-volume\") pod \"payload-processing-external-model-65759456-22sfn\" (UID: \"608043b9-c959-4e55-b909-f810cb6e884a\") " pod="models-as-a-service/payload-processing-external-model-65759456-22sfn"
Nov 03 20:16:42.987294 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:42.985107    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/608043b9-c959-4e55-b909-f810cb6e884a-plugins-config-volume\") pod \"payload-processing-external-model-65759456-22sfn\" (UID: \"608043b9-c959-4e55-b909-f810cb6e884a\") " pod="models-as-a-service/payload-processing-external-model-65759456-22sfn"
Nov 03 20:16:43.018249 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:43.012589    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/608043b9-c959-4e55-b909-f810cb6e884a-provider-credentials\") pod \"payload-processing-external-model-65759456-22sfn\" (UID: \"608043b9-c959-4e55-b909-f810cb6e884a\") " pod="models-as-a-service/payload-processing-external-model-65759456-22sfn"
Nov 03 20:16:43.044229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:43.021607    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lkgw\" (UniqueName: \"kubernetes.io/projected/608043b9-c959-4e55-b909-f810cb6e884a-kube-api-access-7lkgw\") pod \"payload-processing-external-model-65759456-22sfn\" (UID: \"608043b9-c959-4e55-b909-f810cb6e884a\") " pod="models-as-a-service/payload-processing-external-model-65759456-22sfn"
Nov 03 20:16:43.099233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:43.094981    5591 generic.go:356] "Generic (PLEG): container finished" podID="34043cb5-5fdd-4885-9403-5b87877de6cd" containerID="578b912228da7267dbec002a401cd1cb9c8386b5f58041c1d324ac29f0ac3bf9" exitCode=0
Nov 03 20:16:43.099233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:43.095107    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf" event={"ID":"34043cb5-5fdd-4885-9403-5b87877de6cd","Type":"ContainerDied","Data":"578b912228da7267dbec002a401cd1cb9c8386b5f58041c1d324ac29f0ac3bf9"}
Nov 03 20:16:43.099233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:43.095173    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf" event={"ID":"34043cb5-5fdd-4885-9403-5b87877de6cd","Type":"ContainerDied","Data":"1b5162c1f4df1cac8e6b625350f80307a297f598cd0b60e145b0e2b80f6c1b3f"}
Nov 03 20:16:43.099233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:43.095233    5591 scope.go:122] "RemoveContainer" containerID="578b912228da7267dbec002a401cd1cb9c8386b5f58041c1d324ac29f0ac3bf9"
Nov 03 20:16:43.099233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:43.095476    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf"
Nov 03 20:16:43.105801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:43.105525    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65759456-22sfn"
Nov 03 20:16:43.190915 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:43.176101    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf"]
Nov 03 20:16:43.190915 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:43.190822    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-56bdcb5597-tcwmf"]
Nov 03 20:16:43.230620 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:43.230588    5591 scope.go:122] "RemoveContainer" containerID="578b912228da7267dbec002a401cd1cb9c8386b5f58041c1d324ac29f0ac3bf9"
Nov 03 20:16:43.231166 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:16:43.231135    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"578b912228da7267dbec002a401cd1cb9c8386b5f58041c1d324ac29f0ac3bf9\": container with ID starting with 578b912228da7267dbec002a401cd1cb9c8386b5f58041c1d324ac29f0ac3bf9 not found: ID does not exist" containerID="578b912228da7267dbec002a401cd1cb9c8386b5f58041c1d324ac29f0ac3bf9"
Nov 03 20:16:43.231450 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:43.231195    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"578b912228da7267dbec002a401cd1cb9c8386b5f58041c1d324ac29f0ac3bf9"} err="failed to get container status \"578b912228da7267dbec002a401cd1cb9c8386b5f58041c1d324ac29f0ac3bf9\": rpc error: code = NotFound desc = could not find container \"578b912228da7267dbec002a401cd1cb9c8386b5f58041c1d324ac29f0ac3bf9\": container with ID starting with 578b912228da7267dbec002a401cd1cb9c8386b5f58041c1d324ac29f0ac3bf9 not found: ID does not exist"
Nov 03 20:16:43.427687 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:43.427651    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34043cb5-5fdd-4885-9403-5b87877de6cd" path="/var/lib/kubelet/pods/34043cb5-5fdd-4885-9403-5b87877de6cd/volumes"
Nov 03 20:16:43.542772 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:43.542739    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-65759456-22sfn"]
Nov 03 20:16:44.109650 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:44.109601    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65759456-22sfn" event={"ID":"608043b9-c959-4e55-b909-f810cb6e884a","Type":"ContainerStarted","Data":"70c5d3d9a92e57895f8a7d3023d95d855cbd9cf4018ab69d977cd94c04d420c1"}
Nov 03 20:16:44.110096 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:44.109709    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-65759456-22sfn"
Nov 03 20:16:44.110096 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:44.109751    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65759456-22sfn" event={"ID":"608043b9-c959-4e55-b909-f810cb6e884a","Type":"ContainerStarted","Data":"5c4ba948c698200aa2bbab635da182aad135bcd80b0a1e6dcfb2bc05774fe12e"}
Nov 03 20:16:44.140324 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:44.140239    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-65759456-22sfn" podStartSLOduration=4.140217778 podStartE2EDuration="4.140217778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:44.135098294 +0000 UTC m=+2352.907391312" watchObservedRunningTime="2026-11-03 20:16:44.140217778 +0000 UTC m=+2352.912510792"
Nov 03 20:16:45.069564 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:45.069512    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg"
Nov 03 20:16:45.272226 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:45.265537    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv"]
Nov 03 20:16:50.267501 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.267444    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv" podUID="0da83501-fa04-4dea-9ea2-fe64aeb9f15f" containerName="payload-processing" containerID="cri-o://b04f4a6b6f9f8af5a6efcf0c16926d3246ed385f83ad78b064ac12bc3326f2af" gracePeriod=25
Nov 03 20:16:50.581359 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.581329    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv"
Nov 03 20:16:50.618084 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.618051    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww"]
Nov 03 20:16:50.619136 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.619117    5591 cpu_manager.go:414] "containerMap: removing container" podUID="0da83501-fa04-4dea-9ea2-fe64aeb9f15f" containerName="payload-processing"
Nov 03 20:16:50.619136 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.619138    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da83501-fa04-4dea-9ea2-fe64aeb9f15f" containerName="payload-processing"
Nov 03 20:16:50.619343 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.619313    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da83501-fa04-4dea-9ea2-fe64aeb9f15f" containerName="payload-processing"
Nov 03 20:16:50.624035 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.624014    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww"
Nov 03 20:16:50.630031 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.629999    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a0d4ec\"/\"openshift-service-ca.crt\""
Nov 03 20:16:50.630162 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.630087    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a0d4ec\"/\"agc-cred-7548205633a9\""
Nov 03 20:16:50.630232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.630180    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a0d4ec\"/\"kube-root-ca.crt\""
Nov 03 20:16:50.631416 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.630372    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a0d4ec\"/\"routing-overlay\""
Nov 03 20:16:50.631416 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.630486    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a0d4ec\"/\"payload-processing-external-model-e2e--29763338-dockercfg-b9jpc\""
Nov 03 20:16:50.631416 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.630527    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a0d4ec\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a0d4ec\""
Nov 03 20:16:50.634567 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.634546    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww"]
Nov 03 20:16:50.663495 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.663460    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-kube-api-access-7bx2w\" (UniqueName: \"kubernetes.io/projected/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-kube-api-access-7bx2w\") pod \"0da83501-fa04-4dea-9ea2-fe64aeb9f15f\" (UID: \"0da83501-fa04-4dea-9ea2-fe64aeb9f15f\") "
Nov 03 20:16:50.663702 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.663633    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-plugins-config-volume\") pod \"0da83501-fa04-4dea-9ea2-fe64aeb9f15f\" (UID: \"0da83501-fa04-4dea-9ea2-fe64aeb9f15f\") "
Nov 03 20:16:50.663994 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.663971    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-routing-overlay-volume\") pod \"0da83501-fa04-4dea-9ea2-fe64aeb9f15f\" (UID: \"0da83501-fa04-4dea-9ea2-fe64aeb9f15f\") "
Nov 03 20:16:50.664126 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.664026    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-provider-credentials\") pod \"0da83501-fa04-4dea-9ea2-fe64aeb9f15f\" (UID: \"0da83501-fa04-4dea-9ea2-fe64aeb9f15f\") "
Nov 03 20:16:50.664126 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.664051    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-plugins-config-volume" pod "0da83501-fa04-4dea-9ea2-fe64aeb9f15f" (UID: "0da83501-fa04-4dea-9ea2-fe64aeb9f15f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:16:50.664518 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.664492    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-routing-overlay-volume" pod "0da83501-fa04-4dea-9ea2-fe64aeb9f15f" (UID: "0da83501-fa04-4dea-9ea2-fe64aeb9f15f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:16:50.664518 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.664506    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f47c6af8-7866-496c-b1ca-1ebc365884ec-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww\" (UID: \"f47c6af8-7866-496c-b1ca-1ebc365884ec\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww"
Nov 03 20:16:50.664766 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.664676    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f47c6af8-7866-496c-b1ca-1ebc365884ec-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww\" (UID: \"f47c6af8-7866-496c-b1ca-1ebc365884ec\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww"
Nov 03 20:16:50.664766 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.664721    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhslf\" (UniqueName: \"kubernetes.io/projected/f47c6af8-7866-496c-b1ca-1ebc365884ec-kube-api-access-bhslf\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww\" (UID: \"f47c6af8-7866-496c-b1ca-1ebc365884ec\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww"
Nov 03 20:16:50.665023 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.664961    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f47c6af8-7866-496c-b1ca-1ebc365884ec-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww\" (UID: \"f47c6af8-7866-496c-b1ca-1ebc365884ec\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww"
Nov 03 20:16:50.665190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.665159    5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-routing-overlay-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:50.665190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.665179    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:50.666291 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.666265    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-kube-api-access-7bx2w" pod "0da83501-fa04-4dea-9ea2-fe64aeb9f15f" (UID: "0da83501-fa04-4dea-9ea2-fe64aeb9f15f"). InnerVolumeSpecName "kube-api-access-7bx2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:16:50.666678 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.666650    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-provider-credentials" pod "0da83501-fa04-4dea-9ea2-fe64aeb9f15f" (UID: "0da83501-fa04-4dea-9ea2-fe64aeb9f15f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:16:50.766123 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.766082    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f47c6af8-7866-496c-b1ca-1ebc365884ec-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww\" (UID: \"f47c6af8-7866-496c-b1ca-1ebc365884ec\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww"
Nov 03 20:16:50.766393 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.766366    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f47c6af8-7866-496c-b1ca-1ebc365884ec-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww\" (UID: \"f47c6af8-7866-496c-b1ca-1ebc365884ec\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww"
Nov 03 20:16:50.766499 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.766481    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f47c6af8-7866-496c-b1ca-1ebc365884ec-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww\" (UID: \"f47c6af8-7866-496c-b1ca-1ebc365884ec\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww"
Nov 03 20:16:50.766559 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.766527    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhslf\" (UniqueName: \"kubernetes.io/projected/f47c6af8-7866-496c-b1ca-1ebc365884ec-kube-api-access-bhslf\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww\" (UID: \"f47c6af8-7866-496c-b1ca-1ebc365884ec\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww"
Nov 03 20:16:50.766745 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.766725    5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-provider-credentials\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:50.766810 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.766753    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bx2w\" (UniqueName: \"kubernetes.io/projected/0da83501-fa04-4dea-9ea2-fe64aeb9f15f-kube-api-access-7bx2w\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:16:50.766997 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.766970    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f47c6af8-7866-496c-b1ca-1ebc365884ec-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww\" (UID: \"f47c6af8-7866-496c-b1ca-1ebc365884ec\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww"
Nov 03 20:16:50.767244 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.767225    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f47c6af8-7866-496c-b1ca-1ebc365884ec-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww\" (UID: \"f47c6af8-7866-496c-b1ca-1ebc365884ec\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww"
Nov 03 20:16:50.769122 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.769101    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f47c6af8-7866-496c-b1ca-1ebc365884ec-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww\" (UID: \"f47c6af8-7866-496c-b1ca-1ebc365884ec\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww"
Nov 03 20:16:50.779060 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.779036    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhslf\" (UniqueName: \"kubernetes.io/projected/f47c6af8-7866-496c-b1ca-1ebc365884ec-kube-api-access-bhslf\") pod \"payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww\" (UID: \"f47c6af8-7866-496c-b1ca-1ebc365884ec\") " pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww"
Nov 03 20:16:50.976292 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:50.972639    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww"
Nov 03 20:16:51.296149 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:51.296110    5591 generic.go:356] "Generic (PLEG): container finished" podID="0da83501-fa04-4dea-9ea2-fe64aeb9f15f" containerID="b04f4a6b6f9f8af5a6efcf0c16926d3246ed385f83ad78b064ac12bc3326f2af" exitCode=0
Nov 03 20:16:51.296622 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:51.296163    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv"
Nov 03 20:16:51.296622 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:51.296192    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv" event={"ID":"0da83501-fa04-4dea-9ea2-fe64aeb9f15f","Type":"ContainerDied","Data":"b04f4a6b6f9f8af5a6efcf0c16926d3246ed385f83ad78b064ac12bc3326f2af"}
Nov 03 20:16:51.296622 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:51.296267    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv" event={"ID":"0da83501-fa04-4dea-9ea2-fe64aeb9f15f","Type":"ContainerDied","Data":"c8b91ff92402f23f1f15c8d9cffa31ffc461005a0e564d7f2ca2c66fd07d1c02"}
Nov 03 20:16:51.296622 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:51.296291    5591 scope.go:122] "RemoveContainer" containerID="b04f4a6b6f9f8af5a6efcf0c16926d3246ed385f83ad78b064ac12bc3326f2af"
Nov 03 20:16:51.305548 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:51.305524    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww"]
Nov 03 20:16:51.308361 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:16:51.308330    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf47c6af8_7866_496c_b1ca_1ebc365884ec.slice/crio-e6b758d70c7d7fb8aa843ce6cc275b50df70ae96b5ba850b3906859032ff7f13 WatchSource:0}: Error finding container e6b758d70c7d7fb8aa843ce6cc275b50df70ae96b5ba850b3906859032ff7f13: Status 404 returned error can't find the container with id e6b758d70c7d7fb8aa843ce6cc275b50df70ae96b5ba850b3906859032ff7f13
Nov 03 20:16:51.332563 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:51.332541    5591 scope.go:122] "RemoveContainer" containerID="b04f4a6b6f9f8af5a6efcf0c16926d3246ed385f83ad78b064ac12bc3326f2af"
Nov 03 20:16:51.332896 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:16:51.332861    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b04f4a6b6f9f8af5a6efcf0c16926d3246ed385f83ad78b064ac12bc3326f2af\": container with ID starting with b04f4a6b6f9f8af5a6efcf0c16926d3246ed385f83ad78b064ac12bc3326f2af not found: ID does not exist" containerID="b04f4a6b6f9f8af5a6efcf0c16926d3246ed385f83ad78b064ac12bc3326f2af"
Nov 03 20:16:51.333014 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:51.332908    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b04f4a6b6f9f8af5a6efcf0c16926d3246ed385f83ad78b064ac12bc3326f2af"} err="failed to get container status \"b04f4a6b6f9f8af5a6efcf0c16926d3246ed385f83ad78b064ac12bc3326f2af\": rpc error: code = NotFound desc = could not find container \"b04f4a6b6f9f8af5a6efcf0c16926d3246ed385f83ad78b064ac12bc3326f2af\": container with ID starting with b04f4a6b6f9f8af5a6efcf0c16926d3246ed385f83ad78b064ac12bc3326f2af not found: ID does not exist"
Nov 03 20:16:51.358025 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:51.357985    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv"]
Nov 03 20:16:51.358192 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:51.358038    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-dd86zzznv"]
Nov 03 20:16:52.307781 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:52.307742    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww" event={"ID":"f47c6af8-7866-496c-b1ca-1ebc365884ec","Type":"ContainerStarted","Data":"c5681278d737c74ddb16ff8bf740150eb4e65236722ac0e1b9489428212f6b67"}
Nov 03 20:16:52.308344 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:52.307844    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww" event={"ID":"f47c6af8-7866-496c-b1ca-1ebc365884ec","Type":"ContainerStarted","Data":"e6b758d70c7d7fb8aa843ce6cc275b50df70ae96b5ba850b3906859032ff7f13"}
Nov 03 20:16:52.308344 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:52.307998    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww"
Nov 03 20:16:52.329126 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:52.329077    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww" podStartSLOduration=8.329062515 podStartE2EDuration="8.329062515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:52.32739278 +0000 UTC m=+2361.099685814" watchObservedRunningTime="2026-11-03 20:16:52.329062515 +0000 UTC m=+2361.101355545"
Nov 03 20:16:53.353424 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:53.353382    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da83501-fa04-4dea-9ea2-fe64aeb9f15f" path="/var/lib/kubelet/pods/0da83501-fa04-4dea-9ea2-fe64aeb9f15f/volumes"
Nov 03 20:16:55.241040 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:16:55.241008    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-65759456-22sfn"
Nov 03 20:16:58.730077 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:16:58.730033    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:17:03.319409 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:03.319378    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww"
Nov 03 20:17:05.707567 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:05.707529    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg"]
Nov 03 20:17:10.708154 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:10.708110    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg" podUID="c94dcae3-f45e-4e86-a0a7-b22bef29056e" containerName="payload-processing" containerID="cri-o://dd3ebf391e3fbd568def81b19d245298f07f38d0869d7b080fbecade5aa4aae3" gracePeriod=25
Nov 03 20:17:11.028026 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:11.028000    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg"
Nov 03 20:17:11.160260 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:11.160170    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c94dcae3-f45e-4e86-a0a7-b22bef29056e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c94dcae3-f45e-4e86-a0a7-b22bef29056e-provider-credentials\") pod \"c94dcae3-f45e-4e86-a0a7-b22bef29056e\" (UID: \"c94dcae3-f45e-4e86-a0a7-b22bef29056e\") "
Nov 03 20:17:11.160474 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:11.160412    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c94dcae3-f45e-4e86-a0a7-b22bef29056e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c94dcae3-f45e-4e86-a0a7-b22bef29056e-routing-overlay-volume\") pod \"c94dcae3-f45e-4e86-a0a7-b22bef29056e\" (UID: \"c94dcae3-f45e-4e86-a0a7-b22bef29056e\") "
Nov 03 20:17:11.160545 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:11.160473    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c94dcae3-f45e-4e86-a0a7-b22bef29056e-kube-api-access-zprx8\" (UniqueName: \"kubernetes.io/projected/c94dcae3-f45e-4e86-a0a7-b22bef29056e-kube-api-access-zprx8\") pod \"c94dcae3-f45e-4e86-a0a7-b22bef29056e\" (UID: \"c94dcae3-f45e-4e86-a0a7-b22bef29056e\") "
Nov 03 20:17:11.160609 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:11.160587    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c94dcae3-f45e-4e86-a0a7-b22bef29056e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c94dcae3-f45e-4e86-a0a7-b22bef29056e-plugins-config-volume\") pod \"c94dcae3-f45e-4e86-a0a7-b22bef29056e\" (UID: \"c94dcae3-f45e-4e86-a0a7-b22bef29056e\") "
Nov 03 20:17:11.160889 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:11.160828    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c94dcae3-f45e-4e86-a0a7-b22bef29056e-routing-overlay-volume" pod "c94dcae3-f45e-4e86-a0a7-b22bef29056e" (UID: "c94dcae3-f45e-4e86-a0a7-b22bef29056e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:17:11.161172 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:11.161093    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c94dcae3-f45e-4e86-a0a7-b22bef29056e-plugins-config-volume" pod "c94dcae3-f45e-4e86-a0a7-b22bef29056e" (UID: "c94dcae3-f45e-4e86-a0a7-b22bef29056e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:17:11.161514 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:11.161492    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c94dcae3-f45e-4e86-a0a7-b22bef29056e-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:11.161622 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:11.161515    5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c94dcae3-f45e-4e86-a0a7-b22bef29056e-routing-overlay-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:11.162724 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:11.162700    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94dcae3-f45e-4e86-a0a7-b22bef29056e-provider-credentials" pod "c94dcae3-f45e-4e86-a0a7-b22bef29056e" (UID: "c94dcae3-f45e-4e86-a0a7-b22bef29056e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:11.162880 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:11.162861    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94dcae3-f45e-4e86-a0a7-b22bef29056e-kube-api-access-zprx8" pod "c94dcae3-f45e-4e86-a0a7-b22bef29056e" (UID: "c94dcae3-f45e-4e86-a0a7-b22bef29056e"). InnerVolumeSpecName "kube-api-access-zprx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:11.263466 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:11.263414    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zprx8\" (UniqueName: \"kubernetes.io/projected/c94dcae3-f45e-4e86-a0a7-b22bef29056e-kube-api-access-zprx8\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:11.263466 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:11.263437    5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c94dcae3-f45e-4e86-a0a7-b22bef29056e-provider-credentials\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:11.755165 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:11.755071    5591 generic.go:356] "Generic (PLEG): container finished" podID="c94dcae3-f45e-4e86-a0a7-b22bef29056e" containerID="dd3ebf391e3fbd568def81b19d245298f07f38d0869d7b080fbecade5aa4aae3" exitCode=0
Nov 03 20:17:11.755165 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:11.755118    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg"
Nov 03 20:17:11.755689 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:11.755159    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg" event={"ID":"c94dcae3-f45e-4e86-a0a7-b22bef29056e","Type":"ContainerDied","Data":"dd3ebf391e3fbd568def81b19d245298f07f38d0869d7b080fbecade5aa4aae3"}
Nov 03 20:17:11.755689 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:11.755254    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg" event={"ID":"c94dcae3-f45e-4e86-a0a7-b22bef29056e","Type":"ContainerDied","Data":"2eada6776e90ffed23988b09ffc9c340c17e9d971069cda1a9783d85d776d234"}
Nov 03 20:17:11.755689 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:11.755285    5591 scope.go:122] "RemoveContainer" containerID="dd3ebf391e3fbd568def81b19d245298f07f38d0869d7b080fbecade5aa4aae3"
Nov 03 20:17:11.766705 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:11.766672    5591 scope.go:122] "RemoveContainer" containerID="dd3ebf391e3fbd568def81b19d245298f07f38d0869d7b080fbecade5aa4aae3"
Nov 03 20:17:11.766980 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:17:11.766957    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd3ebf391e3fbd568def81b19d245298f07f38d0869d7b080fbecade5aa4aae3\": container with ID starting with dd3ebf391e3fbd568def81b19d245298f07f38d0869d7b080fbecade5aa4aae3 not found: ID does not exist" containerID="dd3ebf391e3fbd568def81b19d245298f07f38d0869d7b080fbecade5aa4aae3"
Nov 03 20:17:11.767048 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:11.766987    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd3ebf391e3fbd568def81b19d245298f07f38d0869d7b080fbecade5aa4aae3"} err="failed to get container status \"dd3ebf391e3fbd568def81b19d245298f07f38d0869d7b080fbecade5aa4aae3\": rpc error: code = NotFound desc = could not find container \"dd3ebf391e3fbd568def81b19d245298f07f38d0869d7b080fbecade5aa4aae3\": container with ID starting with dd3ebf391e3fbd568def81b19d245298f07f38d0869d7b080fbecade5aa4aae3 not found: ID does not exist"
Nov 03 20:17:11.777040 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:11.777011    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg"]
Nov 03 20:17:11.781040 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:11.781017    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-6c646dpzg"]
Nov 03 20:17:13.354284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:13.354247    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c94dcae3-f45e-4e86-a0a7-b22bef29056e" path="/var/lib/kubelet/pods/c94dcae3-f45e-4e86-a0a7-b22bef29056e/volumes"
Nov 03 20:17:21.259754 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:21.259714    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr"]
Nov 03 20:17:21.261490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:21.261468    5591 cpu_manager.go:414] "containerMap: removing container" podUID="c94dcae3-f45e-4e86-a0a7-b22bef29056e" containerName="payload-processing"
Nov 03 20:17:21.261639 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:21.261626    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c94dcae3-f45e-4e86-a0a7-b22bef29056e" containerName="payload-processing"
Nov 03 20:17:21.261941 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:21.261913    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c94dcae3-f45e-4e86-a0a7-b22bef29056e" containerName="payload-processing"
Nov 03 20:17:21.272160 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:21.271678    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr"]
Nov 03 20:17:21.272160 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:21.271800    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr"
Nov 03 20:17:21.277550 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:21.276392    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-51f23d5569d8\""
Nov 03 20:17:21.438092 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:21.438057    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc121b12-4ea4-442f-afb0-d189e332fbfa-provider-credentials\") pod \"payload-processing-external-model-56cf4b877-vxgbr\" (UID: \"cc121b12-4ea4-442f-afb0-d189e332fbfa\") " pod="models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr"
Nov 03 20:17:21.438380 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:21.438108    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc121b12-4ea4-442f-afb0-d189e332fbfa-routing-overlay-volume\") pod \"payload-processing-external-model-56cf4b877-vxgbr\" (UID: \"cc121b12-4ea4-442f-afb0-d189e332fbfa\") " pod="models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr"
Nov 03 20:17:21.438380 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:21.438257    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc121b12-4ea4-442f-afb0-d189e332fbfa-plugins-config-volume\") pod \"payload-processing-external-model-56cf4b877-vxgbr\" (UID: \"cc121b12-4ea4-442f-afb0-d189e332fbfa\") " pod="models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr"
Nov 03 20:17:21.438761 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:21.438730    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp96q\" (UniqueName: \"kubernetes.io/projected/cc121b12-4ea4-442f-afb0-d189e332fbfa-kube-api-access-kp96q\") pod \"payload-processing-external-model-56cf4b877-vxgbr\" (UID: \"cc121b12-4ea4-442f-afb0-d189e332fbfa\") " pod="models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr"
Nov 03 20:17:21.540531 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:21.540496    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp96q\" (UniqueName: \"kubernetes.io/projected/cc121b12-4ea4-442f-afb0-d189e332fbfa-kube-api-access-kp96q\") pod \"payload-processing-external-model-56cf4b877-vxgbr\" (UID: \"cc121b12-4ea4-442f-afb0-d189e332fbfa\") " pod="models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr"
Nov 03 20:17:21.540724 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:21.540620    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc121b12-4ea4-442f-afb0-d189e332fbfa-provider-credentials\") pod \"payload-processing-external-model-56cf4b877-vxgbr\" (UID: \"cc121b12-4ea4-442f-afb0-d189e332fbfa\") " pod="models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr"
Nov 03 20:17:21.540724 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:21.540651    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc121b12-4ea4-442f-afb0-d189e332fbfa-routing-overlay-volume\") pod \"payload-processing-external-model-56cf4b877-vxgbr\" (UID: \"cc121b12-4ea4-442f-afb0-d189e332fbfa\") " pod="models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr"
Nov 03 20:17:21.540724 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:21.540685    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc121b12-4ea4-442f-afb0-d189e332fbfa-plugins-config-volume\") pod \"payload-processing-external-model-56cf4b877-vxgbr\" (UID: \"cc121b12-4ea4-442f-afb0-d189e332fbfa\") " pod="models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr"
Nov 03 20:17:21.541389 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:21.541365    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc121b12-4ea4-442f-afb0-d189e332fbfa-routing-overlay-volume\") pod \"payload-processing-external-model-56cf4b877-vxgbr\" (UID: \"cc121b12-4ea4-442f-afb0-d189e332fbfa\") " pod="models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr"
Nov 03 20:17:21.541550 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:21.541527    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc121b12-4ea4-442f-afb0-d189e332fbfa-plugins-config-volume\") pod \"payload-processing-external-model-56cf4b877-vxgbr\" (UID: \"cc121b12-4ea4-442f-afb0-d189e332fbfa\") " pod="models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr"
Nov 03 20:17:21.543462 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:21.543440    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc121b12-4ea4-442f-afb0-d189e332fbfa-provider-credentials\") pod \"payload-processing-external-model-56cf4b877-vxgbr\" (UID: \"cc121b12-4ea4-442f-afb0-d189e332fbfa\") " pod="models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr"
Nov 03 20:17:21.553548 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:21.553523    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp96q\" (UniqueName: \"kubernetes.io/projected/cc121b12-4ea4-442f-afb0-d189e332fbfa-kube-api-access-kp96q\") pod \"payload-processing-external-model-56cf4b877-vxgbr\" (UID: \"cc121b12-4ea4-442f-afb0-d189e332fbfa\") " pod="models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr"
Nov 03 20:17:21.594091 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:21.594058    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr"
Nov 03 20:17:21.884475 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:17:21.884437    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc121b12_4ea4_442f_afb0_d189e332fbfa.slice/crio-808ceac026908d9f17fb6faadf9daa20ddb0b80c7111ed6ff59c18c1680bd1cd WatchSource:0}: Error finding container 808ceac026908d9f17fb6faadf9daa20ddb0b80c7111ed6ff59c18c1680bd1cd: Status 404 returned error can't find the container with id 808ceac026908d9f17fb6faadf9daa20ddb0b80c7111ed6ff59c18c1680bd1cd
Nov 03 20:17:21.884962 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:21.884937    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr"]
Nov 03 20:17:21.964078 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:21.964044    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr" event={"ID":"cc121b12-4ea4-442f-afb0-d189e332fbfa","Type":"ContainerStarted","Data":"15cb524a90c8ccab44ebfc36fbad0c3e55d33602392fdd29412909f8591a2860"}
Nov 03 20:17:21.964218 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:21.964105    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr" event={"ID":"cc121b12-4ea4-442f-afb0-d189e332fbfa","Type":"ContainerStarted","Data":"808ceac026908d9f17fb6faadf9daa20ddb0b80c7111ed6ff59c18c1680bd1cd"}
Nov 03 20:17:21.964218 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:21.964178    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr"
Nov 03 20:17:21.996087 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:21.996016    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr" podStartSLOduration=0.995995551 podStartE2EDuration="995.995551ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:21.983603599 +0000 UTC m=+2390.755896618" watchObservedRunningTime="2026-11-03 20:17:21.995995551 +0000 UTC m=+2390.768288570"
Nov 03 20:17:28.726562 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:17:28.726525    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:17:32.978985 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:32.978959    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr"
Nov 03 20:17:33.032856 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:33.032823    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-65759456-22sfn"]
Nov 03 20:17:33.712342 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:33.712316    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 20:17:33.712534 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:33.712524    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 20:17:33.990854 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:33.990761    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:17:38.034354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:38.034297    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-65759456-22sfn" podUID="608043b9-c959-4e55-b909-f810cb6e884a" containerName="payload-processing" containerID="cri-o://70c5d3d9a92e57895f8a7d3023d95d855cbd9cf4018ab69d977cd94c04d420c1" gracePeriod=25
Nov 03 20:17:38.172815 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:38.172771    5591 generic.go:356] "Generic (PLEG): container finished" podID="608043b9-c959-4e55-b909-f810cb6e884a" containerID="70c5d3d9a92e57895f8a7d3023d95d855cbd9cf4018ab69d977cd94c04d420c1" exitCode=0
Nov 03 20:17:38.172962 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:38.172851    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65759456-22sfn" event={"ID":"608043b9-c959-4e55-b909-f810cb6e884a","Type":"ContainerDied","Data":"70c5d3d9a92e57895f8a7d3023d95d855cbd9cf4018ab69d977cd94c04d420c1"}
Nov 03 20:17:38.552649 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:38.552612    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65759456-22sfn"
Nov 03 20:17:38.569360 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:38.569334    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/608043b9-c959-4e55-b909-f810cb6e884a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/608043b9-c959-4e55-b909-f810cb6e884a-plugins-config-volume\") pod \"608043b9-c959-4e55-b909-f810cb6e884a\" (UID: \"608043b9-c959-4e55-b909-f810cb6e884a\") "
Nov 03 20:17:38.569507 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:38.569394    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/608043b9-c959-4e55-b909-f810cb6e884a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/608043b9-c959-4e55-b909-f810cb6e884a-provider-credentials\") pod \"608043b9-c959-4e55-b909-f810cb6e884a\" (UID: \"608043b9-c959-4e55-b909-f810cb6e884a\") "
Nov 03 20:17:38.569507 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:38.569484    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/608043b9-c959-4e55-b909-f810cb6e884a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/608043b9-c959-4e55-b909-f810cb6e884a-routing-overlay-volume\") pod \"608043b9-c959-4e55-b909-f810cb6e884a\" (UID: \"608043b9-c959-4e55-b909-f810cb6e884a\") "
Nov 03 20:17:38.569596 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:38.569572    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/608043b9-c959-4e55-b909-f810cb6e884a-kube-api-access-7lkgw\" (UniqueName: \"kubernetes.io/projected/608043b9-c959-4e55-b909-f810cb6e884a-kube-api-access-7lkgw\") pod \"608043b9-c959-4e55-b909-f810cb6e884a\" (UID: \"608043b9-c959-4e55-b909-f810cb6e884a\") "
Nov 03 20:17:38.569797 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:38.569774    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/608043b9-c959-4e55-b909-f810cb6e884a-plugins-config-volume" pod "608043b9-c959-4e55-b909-f810cb6e884a" (UID: "608043b9-c959-4e55-b909-f810cb6e884a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:17:38.569854 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:38.569795    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/608043b9-c959-4e55-b909-f810cb6e884a-routing-overlay-volume" pod "608043b9-c959-4e55-b909-f810cb6e884a" (UID: "608043b9-c959-4e55-b909-f810cb6e884a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:17:38.570702 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:38.570607    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/608043b9-c959-4e55-b909-f810cb6e884a-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:38.570702 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:38.570628    5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/608043b9-c959-4e55-b909-f810cb6e884a-routing-overlay-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:38.572254 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:38.572197    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608043b9-c959-4e55-b909-f810cb6e884a-provider-credentials" pod "608043b9-c959-4e55-b909-f810cb6e884a" (UID: "608043b9-c959-4e55-b909-f810cb6e884a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:38.572347 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:38.572259    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608043b9-c959-4e55-b909-f810cb6e884a-kube-api-access-7lkgw" pod "608043b9-c959-4e55-b909-f810cb6e884a" (UID: "608043b9-c959-4e55-b909-f810cb6e884a"). InnerVolumeSpecName "kube-api-access-7lkgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:38.672389 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:38.672347    5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/608043b9-c959-4e55-b909-f810cb6e884a-provider-credentials\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:38.672389 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:38.672377    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lkgw\" (UniqueName: \"kubernetes.io/projected/608043b9-c959-4e55-b909-f810cb6e884a-kube-api-access-7lkgw\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:39.184050 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:39.184019    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65759456-22sfn"
Nov 03 20:17:39.184587 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:39.184063    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65759456-22sfn" event={"ID":"608043b9-c959-4e55-b909-f810cb6e884a","Type":"ContainerDied","Data":"5c4ba948c698200aa2bbab635da182aad135bcd80b0a1e6dcfb2bc05774fe12e"}
Nov 03 20:17:39.184587 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:39.184127    5591 scope.go:122] "RemoveContainer" containerID="70c5d3d9a92e57895f8a7d3023d95d855cbd9cf4018ab69d977cd94c04d420c1"
Nov 03 20:17:39.210036 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:39.210007    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-65759456-22sfn"]
Nov 03 20:17:39.216127 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:39.216102    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-65759456-22sfn"]
Nov 03 20:17:39.420906 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:39.420869    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="608043b9-c959-4e55-b909-f810cb6e884a" path="/var/lib/kubelet/pods/608043b9-c959-4e55-b909-f810cb6e884a/volumes"
Nov 03 20:17:49.952976 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:49.952938    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88"]
Nov 03 20:17:54.953715 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:54.953672    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88" podUID="f639aa8a-ee24-45d2-a66c-1f363d7c6dca" containerName="payload-processing" containerID="cri-o://26ddbfd0eb68d4adc4dec0755feab73ff6e2657c1ad287c53f609fd60d0a398c" gracePeriod=25
Nov 03 20:17:55.016573 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:17:55.016545    5591 logging.go:55] [core] [Channel #8722 SubChannel #8723]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.230:50052", ServerName: "10.134.0.230:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.230:50052: connect: connection refused"
Nov 03 20:17:55.253600 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.253568    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88"
Nov 03 20:17:55.286916 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.286882    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"]
Nov 03 20:17:55.288667 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.288635    5591 cpu_manager.go:414] "containerMap: removing container" podUID="f639aa8a-ee24-45d2-a66c-1f363d7c6dca" containerName="payload-processing"
Nov 03 20:17:55.288667 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.288666    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f639aa8a-ee24-45d2-a66c-1f363d7c6dca" containerName="payload-processing"
Nov 03 20:17:55.288873 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.288732    5591 cpu_manager.go:414] "containerMap: removing container" podUID="608043b9-c959-4e55-b909-f810cb6e884a" containerName="payload-processing"
Nov 03 20:17:55.288873 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.288740    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="608043b9-c959-4e55-b909-f810cb6e884a" containerName="payload-processing"
Nov 03 20:17:55.288965 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.288946    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="608043b9-c959-4e55-b909-f810cb6e884a" containerName="payload-processing"
Nov 03 20:17:55.289010 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.288977    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f639aa8a-ee24-45d2-a66c-1f363d7c6dca" containerName="payload-processing"
Nov 03 20:17:55.293910 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.293887    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.307857 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.307829    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"]
Nov 03 20:17:55.359226 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.359168    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-kube-api-access-vqqcf\" (UniqueName: \"kubernetes.io/projected/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-kube-api-access-vqqcf\") pod \"f639aa8a-ee24-45d2-a66c-1f363d7c6dca\" (UID: \"f639aa8a-ee24-45d2-a66c-1f363d7c6dca\") "
Nov 03 20:17:55.359578 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.359553    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-routing-overlay-volume\") pod \"f639aa8a-ee24-45d2-a66c-1f363d7c6dca\" (UID: \"f639aa8a-ee24-45d2-a66c-1f363d7c6dca\") "
Nov 03 20:17:55.359734 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.359603    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-provider-credentials\") pod \"f639aa8a-ee24-45d2-a66c-1f363d7c6dca\" (UID: \"f639aa8a-ee24-45d2-a66c-1f363d7c6dca\") "
Nov 03 20:17:55.359734 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.359684    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-plugins-config-volume\") pod \"f639aa8a-ee24-45d2-a66c-1f363d7c6dca\" (UID: \"f639aa8a-ee24-45d2-a66c-1f363d7c6dca\") "
Nov 03 20:17:55.360046 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.360016    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-routing-overlay-volume" pod "f639aa8a-ee24-45d2-a66c-1f363d7c6dca" (UID: "f639aa8a-ee24-45d2-a66c-1f363d7c6dca"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:17:55.360156 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.360113    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-plugins-config-volume" pod "f639aa8a-ee24-45d2-a66c-1f363d7c6dca" (UID: "f639aa8a-ee24-45d2-a66c-1f363d7c6dca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:17:55.360450 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.360431    5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-routing-overlay-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:55.360538 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.360456    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:55.362395 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.362368    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-kube-api-access-vqqcf" pod "f639aa8a-ee24-45d2-a66c-1f363d7c6dca" (UID: "f639aa8a-ee24-45d2-a66c-1f363d7c6dca"). InnerVolumeSpecName "kube-api-access-vqqcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:55.363037 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.363001    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-provider-credentials" pod "f639aa8a-ee24-45d2-a66c-1f363d7c6dca" (UID: "f639aa8a-ee24-45d2-a66c-1f363d7c6dca"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:17:55.461699 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.461668    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-tsrbh\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.461886 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.461707    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-tsrbh\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.461886 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.461799    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tsrbh\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.461981 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.461901    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tsrbh\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.462094 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.462078    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-tsrbh\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.462132 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.462122    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-tsrbh\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.462265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.462244    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2nn9\" (UniqueName: \"kubernetes.io/projected/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-kube-api-access-p2nn9\") pod \"maas-default-gateway-openshift-default-855f9574cf-tsrbh\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.462384 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.462346    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-tsrbh\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.462384 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.462375    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-tsrbh\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.462543 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.462526    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqqcf\" (UniqueName: \"kubernetes.io/projected/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-kube-api-access-vqqcf\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:55.462591 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.462550    5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f639aa8a-ee24-45d2-a66c-1f363d7c6dca-provider-credentials\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:17:55.537820 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.537785    5591 generic.go:356] "Generic (PLEG): container finished" podID="f639aa8a-ee24-45d2-a66c-1f363d7c6dca" containerID="26ddbfd0eb68d4adc4dec0755feab73ff6e2657c1ad287c53f609fd60d0a398c" exitCode=0
Nov 03 20:17:55.537972 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.537835    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88"
Nov 03 20:17:55.537972 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.537872    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88" event={"ID":"f639aa8a-ee24-45d2-a66c-1f363d7c6dca","Type":"ContainerDied","Data":"26ddbfd0eb68d4adc4dec0755feab73ff6e2657c1ad287c53f609fd60d0a398c"}
Nov 03 20:17:55.537972 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.537930    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88" event={"ID":"f639aa8a-ee24-45d2-a66c-1f363d7c6dca","Type":"ContainerDied","Data":"14d50809c2d0757aff53624826241cbf8a18c535605e4fb17df2c5c1ec891c34"}
Nov 03 20:17:55.537972 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.537959    5591 scope.go:122] "RemoveContainer" containerID="26ddbfd0eb68d4adc4dec0755feab73ff6e2657c1ad287c53f609fd60d0a398c"
Nov 03 20:17:55.553644 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.553613    5591 scope.go:122] "RemoveContainer" containerID="26ddbfd0eb68d4adc4dec0755feab73ff6e2657c1ad287c53f609fd60d0a398c"
Nov 03 20:17:55.553983 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:17:55.553963    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26ddbfd0eb68d4adc4dec0755feab73ff6e2657c1ad287c53f609fd60d0a398c\": container with ID starting with 26ddbfd0eb68d4adc4dec0755feab73ff6e2657c1ad287c53f609fd60d0a398c not found: ID does not exist" containerID="26ddbfd0eb68d4adc4dec0755feab73ff6e2657c1ad287c53f609fd60d0a398c"
Nov 03 20:17:55.554068 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.553992    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26ddbfd0eb68d4adc4dec0755feab73ff6e2657c1ad287c53f609fd60d0a398c"} err="failed to get container status \"26ddbfd0eb68d4adc4dec0755feab73ff6e2657c1ad287c53f609fd60d0a398c\": rpc error: code = NotFound desc = could not find container \"26ddbfd0eb68d4adc4dec0755feab73ff6e2657c1ad287c53f609fd60d0a398c\": container with ID starting with 26ddbfd0eb68d4adc4dec0755feab73ff6e2657c1ad287c53f609fd60d0a398c not found: ID does not exist"
Nov 03 20:17:55.564501 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.564462    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-tsrbh\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.564670 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.564568    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-tsrbh\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.564670 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.564585    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-tsrbh\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.565083 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.564787    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tsrbh\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.565083 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.564907    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tsrbh\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.565083 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.565024    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-tsrbh\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.565301 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.565103    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-tsrbh\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.565301 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.565144    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-tsrbh\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.565415 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.565291    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tsrbh\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.565415 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.565325    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2nn9\" (UniqueName: \"kubernetes.io/projected/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-kube-api-access-p2nn9\") pod \"maas-default-gateway-openshift-default-855f9574cf-tsrbh\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.565415 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.565348    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-tsrbh\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.565560 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.565430    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-tsrbh\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.565560 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.565446    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-tsrbh\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.565560 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.565488    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-tsrbh\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.565910 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.565888    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88"]
Nov 03 20:17:55.568318 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.568295    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-tsrbh\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.568670 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.568648    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-tsrbh\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.573624 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.573604    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88"]
Nov 03 20:17:55.578551 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.578528    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-tsrbh\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.578710 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.578614    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2nn9\" (UniqueName: \"kubernetes.io/projected/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-kube-api-access-p2nn9\") pod \"maas-default-gateway-openshift-default-855f9574cf-tsrbh\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:55.625171 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:55.625144    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:56.021442 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:56.021396    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-eb69cb/payload-processing-external-model-e2e-extmod-a-eb69cb-746fpmf88" podUID="f639aa8a-ee24-45d2-a66c-1f363d7c6dca" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.230:50052\" within 1s: context deadline exceeded"
Nov 03 20:17:56.159546 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:17:56.159509    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f5c7f2_56a6_4f8e_ade2_7da70f8ebb7a.slice/crio-9c8456afe0fa29e401f40b15d41180be52bb3dff42e88161e5f0c251a0f044ab WatchSource:0}: Error finding container 9c8456afe0fa29e401f40b15d41180be52bb3dff42e88161e5f0c251a0f044ab: Status 404 returned error can't find the container with id 9c8456afe0fa29e401f40b15d41180be52bb3dff42e88161e5f0c251a0f044ab
Nov 03 20:17:56.160046 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:56.159940    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"]
Nov 03 20:17:56.162653 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:56.162619    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:17:56.162750 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:56.162699    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:17:56.162750 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:56.162733    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:17:56.595157 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:56.595126    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh" event={"ID":"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a","Type":"ContainerStarted","Data":"a196faa4cabf6c21f395ccab70ffa4b179287c34fa6708e3078d53bb9d6dee66"}
Nov 03 20:17:56.595446 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:56.595424    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh" event={"ID":"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a","Type":"ContainerStarted","Data":"9c8456afe0fa29e401f40b15d41180be52bb3dff42e88161e5f0c251a0f044ab"}
Nov 03 20:17:56.623513 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:56.623444    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh" podStartSLOduration=74.623425044 podStartE2EDuration="1m14.623425044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:56.621038975 +0000 UTC m=+2425.393332179" watchObservedRunningTime="2026-11-03 20:17:56.623425044 +0000 UTC m=+2425.395718057"
Nov 03 20:17:56.625548 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:56.625514    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:17:57.352749 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:57.352710    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f639aa8a-ee24-45d2-a66c-1f363d7c6dca" path="/var/lib/kubelet/pods/f639aa8a-ee24-45d2-a66c-1f363d7c6dca/volumes"
Nov 03 20:17:57.626843 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:57.626766    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tsrbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Nov 03 20:17:57.626843 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:57.626818    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh" podUID="f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Nov 03 20:17:58.626082 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:58.626050    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tsrbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Nov 03 20:17:58.626501 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:58.626132    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh" podUID="f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Nov 03 20:17:58.733342 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:17:58.733303    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:17:59.625568 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:59.625522    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tsrbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Nov 03 20:17:59.625764 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:17:59.625600    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh" podUID="f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Nov 03 20:18:00.625627 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:00.625594    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tsrbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Nov 03 20:18:00.626050 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:00.625641    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh" podUID="f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Nov 03 20:18:01.625752 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:01.625712    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tsrbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Nov 03 20:18:01.626164 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:01.625779    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh" podUID="f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Nov 03 20:18:02.626553 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:02.626508    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tsrbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Nov 03 20:18:02.627041 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:02.626600    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh" podUID="f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Nov 03 20:18:03.638245 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:03.636180    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tsrbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body=
Nov 03 20:18:03.638245 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:03.636274    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh" podUID="f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused"
Nov 03 20:18:05.666531 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:05.666469    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-tsrbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:18:05.667084 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:05.666564    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh" podUID="f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:05.806164 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:05.806129    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:18:05.807198 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:05.807152    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:18:05.810991 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:05.810969    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:18:18.558737 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:18.558568    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr"]
Nov 03 20:18:23.559167 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:23.559110    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr" podUID="cc121b12-4ea4-442f-afb0-d189e332fbfa" containerName="payload-processing" containerID="cri-o://15cb524a90c8ccab44ebfc36fbad0c3e55d33602392fdd29412909f8591a2860" gracePeriod=25
Nov 03 20:18:23.877450 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:23.873479    5591 generic.go:356] "Generic (PLEG): container finished" podID="cc121b12-4ea4-442f-afb0-d189e332fbfa" containerID="15cb524a90c8ccab44ebfc36fbad0c3e55d33602392fdd29412909f8591a2860" exitCode=0
Nov 03 20:18:23.877450 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:23.873522    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr" event={"ID":"cc121b12-4ea4-442f-afb0-d189e332fbfa","Type":"ContainerDied","Data":"15cb524a90c8ccab44ebfc36fbad0c3e55d33602392fdd29412909f8591a2860"}
Nov 03 20:18:23.877450 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:23.873585    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr" event={"ID":"cc121b12-4ea4-442f-afb0-d189e332fbfa","Type":"ContainerDied","Data":"808ceac026908d9f17fb6faadf9daa20ddb0b80c7111ed6ff59c18c1680bd1cd"}
Nov 03 20:18:23.877450 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:23.873600    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="808ceac026908d9f17fb6faadf9daa20ddb0b80c7111ed6ff59c18c1680bd1cd"
Nov 03 20:18:23.890718 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:23.890680    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr"
Nov 03 20:18:23.976846 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:23.976797    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc121b12-4ea4-442f-afb0-d189e332fbfa-kube-api-access-kp96q\" (UniqueName: \"kubernetes.io/projected/cc121b12-4ea4-442f-afb0-d189e332fbfa-kube-api-access-kp96q\") pod \"cc121b12-4ea4-442f-afb0-d189e332fbfa\" (UID: \"cc121b12-4ea4-442f-afb0-d189e332fbfa\") "
Nov 03 20:18:23.977057 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:23.976941    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc121b12-4ea4-442f-afb0-d189e332fbfa-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc121b12-4ea4-442f-afb0-d189e332fbfa-routing-overlay-volume\") pod \"cc121b12-4ea4-442f-afb0-d189e332fbfa\" (UID: \"cc121b12-4ea4-442f-afb0-d189e332fbfa\") "
Nov 03 20:18:23.977143 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:23.977108    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc121b12-4ea4-442f-afb0-d189e332fbfa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc121b12-4ea4-442f-afb0-d189e332fbfa-plugins-config-volume\") pod \"cc121b12-4ea4-442f-afb0-d189e332fbfa\" (UID: \"cc121b12-4ea4-442f-afb0-d189e332fbfa\") "
Nov 03 20:18:23.977263 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:23.977244    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc121b12-4ea4-442f-afb0-d189e332fbfa-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc121b12-4ea4-442f-afb0-d189e332fbfa-provider-credentials\") pod \"cc121b12-4ea4-442f-afb0-d189e332fbfa\" (UID: \"cc121b12-4ea4-442f-afb0-d189e332fbfa\") "
Nov 03 20:18:23.977549 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:23.977519    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc121b12-4ea4-442f-afb0-d189e332fbfa-plugins-config-volume" pod "cc121b12-4ea4-442f-afb0-d189e332fbfa" (UID: "cc121b12-4ea4-442f-afb0-d189e332fbfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:23.977684 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:23.977662    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc121b12-4ea4-442f-afb0-d189e332fbfa-routing-overlay-volume" pod "cc121b12-4ea4-442f-afb0-d189e332fbfa" (UID: "cc121b12-4ea4-442f-afb0-d189e332fbfa"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:23.978160 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:23.978137    5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cc121b12-4ea4-442f-afb0-d189e332fbfa-routing-overlay-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:23.978298 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:23.978166    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc121b12-4ea4-442f-afb0-d189e332fbfa-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:23.979902 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:23.979871    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc121b12-4ea4-442f-afb0-d189e332fbfa-provider-credentials" pod "cc121b12-4ea4-442f-afb0-d189e332fbfa" (UID: "cc121b12-4ea4-442f-afb0-d189e332fbfa"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:23.979992 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:23.979930    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc121b12-4ea4-442f-afb0-d189e332fbfa-kube-api-access-kp96q" pod "cc121b12-4ea4-442f-afb0-d189e332fbfa" (UID: "cc121b12-4ea4-442f-afb0-d189e332fbfa"). InnerVolumeSpecName "kube-api-access-kp96q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:24.080139 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:24.080100    5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cc121b12-4ea4-442f-afb0-d189e332fbfa-provider-credentials\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:24.080139 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:24.080133    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp96q\" (UniqueName: \"kubernetes.io/projected/cc121b12-4ea4-442f-afb0-d189e332fbfa-kube-api-access-kp96q\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:24.935514 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:24.935481    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr"
Nov 03 20:18:24.965767 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:24.965739    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr"]
Nov 03 20:18:24.969825 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:24.969804    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-56cf4b877-vxgbr"]
Nov 03 20:18:25.126242 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:25.126176    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:18:25.353508 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:25.353479    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc121b12-4ea4-442f-afb0-d189e332fbfa" path="/var/lib/kubelet/pods/cc121b12-4ea4-442f-afb0-d189e332fbfa/volumes"
Nov 03 20:18:28.871089 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:18:28.871051    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:18:34.671015 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:34.670977    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"]
Nov 03 20:18:34.671498 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:34.671259    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv" podUID="53f1ccac-bfc0-4b66-a445-86b3c40afdfe" containerName="istio-proxy" containerID="cri-o://c7f47954a4d434316a07bc4297b472ebb2324292c4dd68766b961faecbfd691a" gracePeriod=30
Nov 03 20:18:38.636348 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:38.636315    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:18:38.636791 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:38.636380    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv" podUID="53f1ccac-bfc0-4b66-a445-86b3c40afdfe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:18:40.033467 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.033438    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 20:18:40.071575 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.071536    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"]
Nov 03 20:18:40.072490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.072469    5591 cpu_manager.go:414] "containerMap: removing container" podUID="53f1ccac-bfc0-4b66-a445-86b3c40afdfe" containerName="istio-proxy"
Nov 03 20:18:40.072490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.072487    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53f1ccac-bfc0-4b66-a445-86b3c40afdfe" containerName="istio-proxy"
Nov 03 20:18:40.072694 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.072531    5591 cpu_manager.go:414] "containerMap: removing container" podUID="cc121b12-4ea4-442f-afb0-d189e332fbfa" containerName="payload-processing"
Nov 03 20:18:40.072694 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.072536    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc121b12-4ea4-442f-afb0-d189e332fbfa" containerName="payload-processing"
Nov 03 20:18:40.072694 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.072665    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc121b12-4ea4-442f-afb0-d189e332fbfa" containerName="payload-processing"
Nov 03 20:18:40.072694 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.072682    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="53f1ccac-bfc0-4b66-a445-86b3c40afdfe" containerName="istio-proxy"
Nov 03 20:18:40.076980 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.076959    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.089998 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.089018    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"]
Nov 03 20:18:40.191484 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.191456    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-kube-api-access-8vbp8\" (UniqueName: \"kubernetes.io/projected/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-kube-api-access-8vbp8\") pod \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") "
Nov 03 20:18:40.191696 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.191520    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-podinfo\") pod \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") "
Nov 03 20:18:40.191696 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.191588    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-data\") pod \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") "
Nov 03 20:18:40.191696 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.191660    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-envoy\") pod \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") "
Nov 03 20:18:40.191889 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.191721    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-workload-socket\") pod \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") "
Nov 03 20:18:40.191889 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.191745    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-credential-socket\") pod \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") "
Nov 03 20:18:40.191889 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.191774    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-token\" (UniqueName: \"kubernetes.io/projected/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-token\") pod \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") "
Nov 03 20:18:40.192027 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.191902    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-workload-certs\") pod \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") "
Nov 03 20:18:40.192027 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.191937    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istiod-ca-cert\") pod \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\" (UID: \"53f1ccac-bfc0-4b66-a445-86b3c40afdfe\") "
Nov 03 20:18:40.192027 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.191936    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-data" pod "53f1ccac-bfc0-4b66-a445-86b3c40afdfe" (UID: "53f1ccac-bfc0-4b66-a445-86b3c40afdfe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:40.192027 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.191967    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-workload-socket" pod "53f1ccac-bfc0-4b66-a445-86b3c40afdfe" (UID: "53f1ccac-bfc0-4b66-a445-86b3c40afdfe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:40.192247 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.192072    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-credential-socket" pod "53f1ccac-bfc0-4b66-a445-86b3c40afdfe" (UID: "53f1ccac-bfc0-4b66-a445-86b3c40afdfe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:40.192247 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.192179    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-workload-certs" pod "53f1ccac-bfc0-4b66-a445-86b3c40afdfe" (UID: "53f1ccac-bfc0-4b66-a445-86b3c40afdfe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:40.192539 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.192470    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istiod-ca-cert" pod "53f1ccac-bfc0-4b66-a445-86b3c40afdfe" (UID: "53f1ccac-bfc0-4b66-a445-86b3c40afdfe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:18:40.192539 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.192495    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-kqcwg\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.192801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.192779    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-kqcwg\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.192921 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.192894    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-kqcwg\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.192993 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.192961    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv5zr\" (UniqueName: \"kubernetes.io/projected/53641647-e68c-4a85-8d1a-fa903d2acdad-kube-api-access-pv5zr\") pod \"maas-default-gateway-openshift-default-855f9574cf-kqcwg\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.193057 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.193026    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-kqcwg\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.193117 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.193083    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-kqcwg\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.193227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.193188    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-kqcwg\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.193478 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.193451    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-kqcwg\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.193729 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.193698    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53641647-e68c-4a85-8d1a-fa903d2acdad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-kqcwg\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.193913 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.193890    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:40.194031 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.193920    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:40.194031 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.193937    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:40.194031 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.193950    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:40.194031 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.193962    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:40.194528 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.194507    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-podinfo" pod "53f1ccac-bfc0-4b66-a445-86b3c40afdfe" (UID: "53f1ccac-bfc0-4b66-a445-86b3c40afdfe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:18:40.194528 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.194517    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-envoy" pod "53f1ccac-bfc0-4b66-a445-86b3c40afdfe" (UID: "53f1ccac-bfc0-4b66-a445-86b3c40afdfe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:18:40.194626 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.194593    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-kube-api-access-8vbp8" pod "53f1ccac-bfc0-4b66-a445-86b3c40afdfe" (UID: "53f1ccac-bfc0-4b66-a445-86b3c40afdfe"). InnerVolumeSpecName "kube-api-access-8vbp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:40.194660 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.194631    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-token" pod "53f1ccac-bfc0-4b66-a445-86b3c40afdfe" (UID: "53f1ccac-bfc0-4b66-a445-86b3c40afdfe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:18:40.257110 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.257011    5591 generic.go:356] "Generic (PLEG): container finished" podID="53f1ccac-bfc0-4b66-a445-86b3c40afdfe" containerID="c7f47954a4d434316a07bc4297b472ebb2324292c4dd68766b961faecbfd691a" exitCode=0
Nov 03 20:18:40.257110 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.257083    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"
Nov 03 20:18:40.257395 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.257120    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv" event={"ID":"53f1ccac-bfc0-4b66-a445-86b3c40afdfe","Type":"ContainerDied","Data":"c7f47954a4d434316a07bc4297b472ebb2324292c4dd68766b961faecbfd691a"}
Nov 03 20:18:40.257395 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.257172    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv" event={"ID":"53f1ccac-bfc0-4b66-a445-86b3c40afdfe","Type":"ContainerDied","Data":"6e7cd03665329ba0df26ff8db19a6db1a169e9e92654ef504a4f5837e556bcab"}
Nov 03 20:18:40.257395 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.257243    5591 scope.go:122] "RemoveContainer" containerID="c7f47954a4d434316a07bc4297b472ebb2324292c4dd68766b961faecbfd691a"
Nov 03 20:18:40.270366 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.270348    5591 scope.go:122] "RemoveContainer" containerID="c7f47954a4d434316a07bc4297b472ebb2324292c4dd68766b961faecbfd691a"
Nov 03 20:18:40.270625 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:18:40.270608    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7f47954a4d434316a07bc4297b472ebb2324292c4dd68766b961faecbfd691a\": container with ID starting with c7f47954a4d434316a07bc4297b472ebb2324292c4dd68766b961faecbfd691a not found: ID does not exist" containerID="c7f47954a4d434316a07bc4297b472ebb2324292c4dd68766b961faecbfd691a"
Nov 03 20:18:40.270670 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.270632    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7f47954a4d434316a07bc4297b472ebb2324292c4dd68766b961faecbfd691a"} err="failed to get container status \"c7f47954a4d434316a07bc4297b472ebb2324292c4dd68766b961faecbfd691a\": rpc error: code = NotFound desc = could not find container \"c7f47954a4d434316a07bc4297b472ebb2324292c4dd68766b961faecbfd691a\": container with ID starting with c7f47954a4d434316a07bc4297b472ebb2324292c4dd68766b961faecbfd691a not found: ID does not exist"
Nov 03 20:18:40.284879 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.284852    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"]
Nov 03 20:18:40.288956 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.288935    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-bsqpv"]
Nov 03 20:18:40.295354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.295331    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53641647-e68c-4a85-8d1a-fa903d2acdad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-kqcwg\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.295482 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.295419    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-kqcwg\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.295559 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.295494    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-kqcwg\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.295559 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.295552    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-kqcwg\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.295735 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.295595    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv5zr\" (UniqueName: \"kubernetes.io/projected/53641647-e68c-4a85-8d1a-fa903d2acdad-kube-api-access-pv5zr\") pod \"maas-default-gateway-openshift-default-855f9574cf-kqcwg\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.295735 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.295641    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-kqcwg\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.295735 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.295690    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-kqcwg\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.295904 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.295783    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-kqcwg\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.295904 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.295840    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-kqcwg\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.295904 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.295865    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-kqcwg\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.296115 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.295985    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vbp8\" (UniqueName: \"kubernetes.io/projected/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-kube-api-access-8vbp8\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:40.296115 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.296001    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:40.296115 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.296016    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:40.296115 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.296029    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53f1ccac-bfc0-4b66-a445-86b3c40afdfe-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:18:40.296115 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.296081    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-kqcwg\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.296452 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.296131    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53641647-e68c-4a85-8d1a-fa903d2acdad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-kqcwg\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.296452 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.296380    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-kqcwg\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.296747 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.296506    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-kqcwg\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.298357 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.298337    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-kqcwg\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.298695 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.298673    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-kqcwg\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.314616 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.314595    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-kqcwg\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.314818 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.314800    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv5zr\" (UniqueName: \"kubernetes.io/projected/53641647-e68c-4a85-8d1a-fa903d2acdad-kube-api-access-pv5zr\") pod \"maas-default-gateway-openshift-default-855f9574cf-kqcwg\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.404359 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.404324    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:40.749228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.743078    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"]
Nov 03 20:18:40.773763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.772950    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:40.773763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.773055    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:40.773763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:40.773099    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:18:41.267491 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:41.267450    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg" event={"ID":"53641647-e68c-4a85-8d1a-fa903d2acdad","Type":"ContainerStarted","Data":"d3f75afa34deb64b495136125a529cffcb613a97568ccb7074588bfb377923ec"}
Nov 03 20:18:41.267891 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:41.267533    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg" event={"ID":"53641647-e68c-4a85-8d1a-fa903d2acdad","Type":"ContainerStarted","Data":"2cbe803389ab3ce699eba2e303e5d556472b064359477c28bfba4c0593a6179b"}
Nov 03 20:18:41.302719 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:41.302653    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg" podStartSLOduration=119.302633434 podStartE2EDuration="1m59.302633434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:41.296076869 +0000 UTC m=+2470.068369900" watchObservedRunningTime="2026-11-03 20:18:41.302633434 +0000 UTC m=+2470.074926463"
Nov 03 20:18:41.354108 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:41.354075    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53f1ccac-bfc0-4b66-a445-86b3c40afdfe" path="/var/lib/kubelet/pods/53f1ccac-bfc0-4b66-a445-86b3c40afdfe/volumes"
Nov 03 20:18:41.404888 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:41.404847    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:41.406436 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:41.406406    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kqcwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Nov 03 20:18:41.406623 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:41.406460    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg" podUID="53641647-e68c-4a85-8d1a-fa903d2acdad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Nov 03 20:18:42.405109 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:42.405070    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kqcwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Nov 03 20:18:42.405609 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:42.405143    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg" podUID="53641647-e68c-4a85-8d1a-fa903d2acdad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Nov 03 20:18:43.405281 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:43.405240    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kqcwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Nov 03 20:18:43.405982 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:43.405867    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg" podUID="53641647-e68c-4a85-8d1a-fa903d2acdad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Nov 03 20:18:44.406918 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:44.406879    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kqcwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Nov 03 20:18:44.407655 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:44.407606    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg" podUID="53641647-e68c-4a85-8d1a-fa903d2acdad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Nov 03 20:18:45.405969 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:45.405925    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kqcwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Nov 03 20:18:45.406313 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:45.406278    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg" podUID="53641647-e68c-4a85-8d1a-fa903d2acdad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Nov 03 20:18:46.448232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:46.435056    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kqcwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Nov 03 20:18:46.448232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:46.435186    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg" podUID="53641647-e68c-4a85-8d1a-fa903d2acdad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Nov 03 20:18:47.424255 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:47.423017    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kqcwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Nov 03 20:18:47.424255 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:47.423081    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg" podUID="53641647-e68c-4a85-8d1a-fa903d2acdad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Nov 03 20:18:47.607233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:47.582323    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:47.702444 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:47.702317    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:48.101486 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:48.100321    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": context deadline exceeded"
Nov 03 20:18:48.409291 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:48.408660    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kqcwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Nov 03 20:18:48.409291 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:48.408764    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg" podUID="53641647-e68c-4a85-8d1a-fa903d2acdad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Nov 03 20:18:49.405079 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:49.405038    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kqcwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body=
Nov 03 20:18:49.405535 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:49.405099    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg" podUID="53641647-e68c-4a85-8d1a-fa903d2acdad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused"
Nov 03 20:18:50.897835 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:50.897785    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk"]
Nov 03 20:18:50.905498 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:50.905470    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk"
Nov 03 20:18:50.916547 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:50.916515    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e1546\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0e1546\""
Nov 03 20:18:50.916775 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:50.916750    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e1546\"/\"openshift-service-ca.crt\""
Nov 03 20:18:50.916967 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:50.916942    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e1546\"/\"routing-overlay\""
Nov 03 20:18:50.917067 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:50.916876    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e1546\"/\"kube-root-ca.crt\""
Nov 03 20:18:50.923498 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:50.923474    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk"]
Nov 03 20:18:50.946827 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:50.946793    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0e1546\"/\"agc-cred-73ce6d5c12e7\""
Nov 03 20:18:50.965895 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:50.965829    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk\" (UID: \"3e193fbd-3787-48cc-a6ad-dcfba1ef0249\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk"
Nov 03 20:18:50.966181 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:50.965920    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk\" (UID: \"3e193fbd-3787-48cc-a6ad-dcfba1ef0249\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk"
Nov 03 20:18:50.966181 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:50.965970    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v55ln\" (UniqueName: \"kubernetes.io/projected/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-kube-api-access-v55ln\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk\" (UID: \"3e193fbd-3787-48cc-a6ad-dcfba1ef0249\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk"
Nov 03 20:18:50.966181 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:50.966174    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk\" (UID: \"3e193fbd-3787-48cc-a6ad-dcfba1ef0249\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk"
Nov 03 20:18:51.068572 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:51.068525    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk\" (UID: \"3e193fbd-3787-48cc-a6ad-dcfba1ef0249\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk"
Nov 03 20:18:51.068832 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:51.068811    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk\" (UID: \"3e193fbd-3787-48cc-a6ad-dcfba1ef0249\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk"
Nov 03 20:18:51.068924 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:51.068889    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk\" (UID: \"3e193fbd-3787-48cc-a6ad-dcfba1ef0249\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk"
Nov 03 20:18:51.068998 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:51.068933    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v55ln\" (UniqueName: \"kubernetes.io/projected/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-kube-api-access-v55ln\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk\" (UID: \"3e193fbd-3787-48cc-a6ad-dcfba1ef0249\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk"
Nov 03 20:18:51.069716 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:51.069686    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk\" (UID: \"3e193fbd-3787-48cc-a6ad-dcfba1ef0249\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk"
Nov 03 20:18:51.070438 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:51.070405    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk\" (UID: \"3e193fbd-3787-48cc-a6ad-dcfba1ef0249\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk"
Nov 03 20:18:51.072817 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:51.071912    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk\" (UID: \"3e193fbd-3787-48cc-a6ad-dcfba1ef0249\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk"
Nov 03 20:18:51.115529 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:51.115492    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v55ln\" (UniqueName: \"kubernetes.io/projected/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-kube-api-access-v55ln\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk\" (UID: \"3e193fbd-3787-48cc-a6ad-dcfba1ef0249\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk"
Nov 03 20:18:51.232864 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:51.232718    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk"
Nov 03 20:18:51.459195 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:51.458615    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kqcwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:18:51.459195 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:51.458678    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg" podUID="53641647-e68c-4a85-8d1a-fa903d2acdad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:18:51.905993 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:51.905956    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:51.906803 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:51.906494    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:51.908490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:51.908455    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:18:51.945142 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:51.945110    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk"]
Nov 03 20:18:51.945838 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:18:51.945811    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e193fbd_3787_48cc_a6ad_dcfba1ef0249.slice/crio-214bb90830126bc0d1282038e4e9eac51823c298bdafca6fdae2412fd07d47b4 WatchSource:0}: Error finding container 214bb90830126bc0d1282038e4e9eac51823c298bdafca6fdae2412fd07d47b4: Status 404 returned error can't find the container with id 214bb90830126bc0d1282038e4e9eac51823c298bdafca6fdae2412fd07d47b4
Nov 03 20:18:52.737896 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:52.737771    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk" event={"ID":"3e193fbd-3787-48cc-a6ad-dcfba1ef0249","Type":"ContainerStarted","Data":"f9f9bba42fe1c164c3ff9e521859e4c5bb64e502a1e4c4a347c9d7a66cf8a7cc"}
Nov 03 20:18:52.737896 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:52.737849    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk" event={"ID":"3e193fbd-3787-48cc-a6ad-dcfba1ef0249","Type":"ContainerStarted","Data":"214bb90830126bc0d1282038e4e9eac51823c298bdafca6fdae2412fd07d47b4"}
Nov 03 20:18:52.738452 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:52.738432    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk"
Nov 03 20:18:52.776914 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:18:52.776855    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk" podStartSLOduration=2.776840142 podStartE2EDuration="2.776840142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:52.774950684 +0000 UTC m=+2481.547243769" watchObservedRunningTime="2026-11-03 20:18:52.776840142 +0000 UTC m=+2481.549133160"
Nov 03 20:18:58.728615 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:18:58.728568    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:19:03.752601 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:19:03.752570    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk"
Nov 03 20:19:09.426285 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:19:09.426249    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:19:28.728545 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:19:28.728503    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:19:46.525356 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:19:46.508318    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:46.552062 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:19:46.552000    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:46.603196 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:19:46.597694    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:46.771503 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:19:46.771462    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:19:58.744339 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:19:58.744294    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:20:04.757501 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:04.757464    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:20:28.941119 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:20:28.941073    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:20:30.241287 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:30.241252    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk"]
Nov 03 20:20:35.241747 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.241702    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk" podUID="3e193fbd-3787-48cc-a6ad-dcfba1ef0249" containerName="payload-processing" containerID="cri-o://f9f9bba42fe1c164c3ff9e521859e4c5bb64e502a1e4c4a347c9d7a66cf8a7cc" gracePeriod=25
Nov 03 20:20:35.658904 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.658869    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk"
Nov 03 20:20:35.672493 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.672462    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-7ec3de/e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx"]
Nov 03 20:20:35.673636 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.673617    5591 cpu_manager.go:414] "containerMap: removing container" podUID="3e193fbd-3787-48cc-a6ad-dcfba1ef0249" containerName="payload-processing"
Nov 03 20:20:35.673707 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.673638    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e193fbd-3787-48cc-a6ad-dcfba1ef0249" containerName="payload-processing"
Nov 03 20:20:35.673783 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.673773    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e193fbd-3787-48cc-a6ad-dcfba1ef0249" containerName="payload-processing"
Nov 03 20:20:35.677898 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.677877    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-7ec3de/e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx"
Nov 03 20:20:35.683442 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.683116    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-7ec3de\"/\"kube-root-ca.crt\""
Nov 03 20:20:35.684701 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.684671    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-7ec3de\"/\"default-dockercfg-bdm72\""
Nov 03 20:20:35.684856 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.684770    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-7ec3de\"/\"e2e-usage-sim-5a5376-cert\""
Nov 03 20:20:35.685720 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.685698    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-7ec3de\"/\"openshift-service-ca.crt\""
Nov 03 20:20:35.687268 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.687198    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-7ec3de/e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx"]
Nov 03 20:20:35.754790 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.754754    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-provider-credentials\") pod \"3e193fbd-3787-48cc-a6ad-dcfba1ef0249\" (UID: \"3e193fbd-3787-48cc-a6ad-dcfba1ef0249\") "
Nov 03 20:20:35.755015 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.754870    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-plugins-config-volume\") pod \"3e193fbd-3787-48cc-a6ad-dcfba1ef0249\" (UID: \"3e193fbd-3787-48cc-a6ad-dcfba1ef0249\") "
Nov 03 20:20:35.755104 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.755034    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-kube-api-access-v55ln\" (UniqueName: \"kubernetes.io/projected/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-kube-api-access-v55ln\") pod \"3e193fbd-3787-48cc-a6ad-dcfba1ef0249\" (UID: \"3e193fbd-3787-48cc-a6ad-dcfba1ef0249\") "
Nov 03 20:20:35.755385 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.755361    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-routing-overlay-volume\") pod \"3e193fbd-3787-48cc-a6ad-dcfba1ef0249\" (UID: \"3e193fbd-3787-48cc-a6ad-dcfba1ef0249\") "
Nov 03 20:20:35.762224 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.755671    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-plugins-config-volume" pod "3e193fbd-3787-48cc-a6ad-dcfba1ef0249" (UID: "3e193fbd-3787-48cc-a6ad-dcfba1ef0249"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:35.762387 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.756167    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-routing-overlay-volume" pod "3e193fbd-3787-48cc-a6ad-dcfba1ef0249" (UID: "3e193fbd-3787-48cc-a6ad-dcfba1ef0249"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:35.762387 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.759479    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-provider-credentials" pod "3e193fbd-3787-48cc-a6ad-dcfba1ef0249" (UID: "3e193fbd-3787-48cc-a6ad-dcfba1ef0249"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:35.762543 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.762517    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-kube-api-access-v55ln" pod "3e193fbd-3787-48cc-a6ad-dcfba1ef0249" (UID: "3e193fbd-3787-48cc-a6ad-dcfba1ef0249"). InnerVolumeSpecName "kube-api-access-v55ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:35.762735 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.762705    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9frmk\" (UniqueName: \"kubernetes.io/projected/d7710d70-6f64-455a-93de-8ef183cede56-kube-api-access-9frmk\") pod \"e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx\" (UID: \"d7710d70-6f64-455a-93de-8ef183cede56\") " pod="e2e-extmod-shr-user-7ec3de/e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx"
Nov 03 20:20:35.763165 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.763141    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d7710d70-6f64-455a-93de-8ef183cede56-tls\") pod \"e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx\" (UID: \"d7710d70-6f64-455a-93de-8ef183cede56\") " pod="e2e-extmod-shr-user-7ec3de/e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx"
Nov 03 20:20:35.763337 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.763318    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v55ln\" (UniqueName: \"kubernetes.io/projected/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-kube-api-access-v55ln\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:35.763400 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.763345    5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-routing-overlay-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:35.763400 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.763362    5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-provider-credentials\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:35.763400 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.763377    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e193fbd-3787-48cc-a6ad-dcfba1ef0249-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:35.864772 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.864668    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d7710d70-6f64-455a-93de-8ef183cede56-tls\") pod \"e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx\" (UID: \"d7710d70-6f64-455a-93de-8ef183cede56\") " pod="e2e-extmod-shr-user-7ec3de/e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx"
Nov 03 20:20:35.864970 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.864810    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9frmk\" (UniqueName: \"kubernetes.io/projected/d7710d70-6f64-455a-93de-8ef183cede56-kube-api-access-9frmk\") pod \"e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx\" (UID: \"d7710d70-6f64-455a-93de-8ef183cede56\") " pod="e2e-extmod-shr-user-7ec3de/e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx"
Nov 03 20:20:35.867472 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.867450    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d7710d70-6f64-455a-93de-8ef183cede56-tls\") pod \"e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx\" (UID: \"d7710d70-6f64-455a-93de-8ef183cede56\") " pod="e2e-extmod-shr-user-7ec3de/e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx"
Nov 03 20:20:35.877277 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.877250    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9frmk\" (UniqueName: \"kubernetes.io/projected/d7710d70-6f64-455a-93de-8ef183cede56-kube-api-access-9frmk\") pod \"e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx\" (UID: \"d7710d70-6f64-455a-93de-8ef183cede56\") " pod="e2e-extmod-shr-user-7ec3de/e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx"
Nov 03 20:20:35.994751 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:35.994716    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-7ec3de/e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx"
Nov 03 20:20:36.319934 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:36.319900    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-7ec3de/e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx"]
Nov 03 20:20:36.320640 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:20:36.320615    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7710d70_6f64_455a_93de_8ef183cede56.slice/crio-ac7e98d8386c7abc6b8e4288b7e499cf6939fc08858c6ddbfe8e3bcb998ef48c WatchSource:0}: Error finding container ac7e98d8386c7abc6b8e4288b7e499cf6939fc08858c6ddbfe8e3bcb998ef48c: Status 404 returned error can't find the container with id ac7e98d8386c7abc6b8e4288b7e499cf6939fc08858c6ddbfe8e3bcb998ef48c
Nov 03 20:20:36.384829 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:36.384777    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-7ec3de/e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx" event={"ID":"d7710d70-6f64-455a-93de-8ef183cede56","Type":"ContainerStarted","Data":"ac7e98d8386c7abc6b8e4288b7e499cf6939fc08858c6ddbfe8e3bcb998ef48c"}
Nov 03 20:20:36.387113 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:36.387079    5591 generic.go:356] "Generic (PLEG): container finished" podID="3e193fbd-3787-48cc-a6ad-dcfba1ef0249" containerID="f9f9bba42fe1c164c3ff9e521859e4c5bb64e502a1e4c4a347c9d7a66cf8a7cc" exitCode=0
Nov 03 20:20:36.387325 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:36.387272    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk" event={"ID":"3e193fbd-3787-48cc-a6ad-dcfba1ef0249","Type":"ContainerDied","Data":"f9f9bba42fe1c164c3ff9e521859e4c5bb64e502a1e4c4a347c9d7a66cf8a7cc"}
Nov 03 20:20:36.387402 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:36.387351    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk" event={"ID":"3e193fbd-3787-48cc-a6ad-dcfba1ef0249","Type":"ContainerDied","Data":"214bb90830126bc0d1282038e4e9eac51823c298bdafca6fdae2412fd07d47b4"}
Nov 03 20:20:36.387402 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:36.387392    5591 scope.go:122] "RemoveContainer" containerID="f9f9bba42fe1c164c3ff9e521859e4c5bb64e502a1e4c4a347c9d7a66cf8a7cc"
Nov 03 20:20:36.387738 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:36.387602    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk"
Nov 03 20:20:36.408055 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:36.408029    5591 scope.go:122] "RemoveContainer" containerID="f9f9bba42fe1c164c3ff9e521859e4c5bb64e502a1e4c4a347c9d7a66cf8a7cc"
Nov 03 20:20:36.408385 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:20:36.408360    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9f9bba42fe1c164c3ff9e521859e4c5bb64e502a1e4c4a347c9d7a66cf8a7cc\": container with ID starting with f9f9bba42fe1c164c3ff9e521859e4c5bb64e502a1e4c4a347c9d7a66cf8a7cc not found: ID does not exist" containerID="f9f9bba42fe1c164c3ff9e521859e4c5bb64e502a1e4c4a347c9d7a66cf8a7cc"
Nov 03 20:20:36.408476 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:36.408399    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f9bba42fe1c164c3ff9e521859e4c5bb64e502a1e4c4a347c9d7a66cf8a7cc"} err="failed to get container status \"f9f9bba42fe1c164c3ff9e521859e4c5bb64e502a1e4c4a347c9d7a66cf8a7cc\": rpc error: code = NotFound desc = could not find container \"f9f9bba42fe1c164c3ff9e521859e4c5bb64e502a1e4c4a347c9d7a66cf8a7cc\": container with ID starting with f9f9bba42fe1c164c3ff9e521859e4c5bb64e502a1e4c4a347c9d7a66cf8a7cc not found: ID does not exist"
Nov 03 20:20:36.423582 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:36.423554    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk"]
Nov 03 20:20:36.428249 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:36.428222    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-86462czlk"]
Nov 03 20:20:37.352594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:37.352559    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e193fbd-3787-48cc-a6ad-dcfba1ef0249" path="/var/lib/kubelet/pods/3e193fbd-3787-48cc-a6ad-dcfba1ef0249/volumes"
Nov 03 20:20:37.398537 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:37.398499    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-7ec3de/e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx" event={"ID":"d7710d70-6f64-455a-93de-8ef183cede56","Type":"ContainerStarted","Data":"2bc2389cb49d89d35a7d9ac6677f95704cf1628e0351ec8b6cf3d5a36771a77a"}
Nov 03 20:20:37.398813 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:37.398792    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-7ec3de/e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx"
Nov 03 20:20:37.403692 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:37.403666    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-7ec3de/e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx"
Nov 03 20:20:37.420288 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:37.420240    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-7ec3de/e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx" podStartSLOduration=2.420227385 podStartE2EDuration="2.420227385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:37.416102384 +0000 UTC m=+2586.188395402" watchObservedRunningTime="2026-11-03 20:20:37.420227385 +0000 UTC m=+2586.192520394"
Nov 03 20:20:42.590278 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:42.588563    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:20:42.769355 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:42.769309    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:42.773301 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:42.773265    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25"]
Nov 03 20:20:43.042290 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:43.013650    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25"
Nov 03 20:20:43.061371 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:43.061302    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25"]
Nov 03 20:20:43.170985 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:43.154665    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"agc-cred-ccc316971923\""
Nov 03 20:20:43.170985 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:43.155232    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"payload-processing-external-model-plugins-e2e-extmod-a-33e267\""
Nov 03 20:20:43.170985 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:43.155352    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"agc-cred-53243bacc083\""
Nov 03 20:20:43.170985 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:43.155421    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"agc-cred-e5af0b75dc63\""
Nov 03 20:20:43.170985 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:43.155444    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"payload-processing-external-model-e2e--aa51d60f-dockercfg-wjs86\""
Nov 03 20:20:43.170985 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:43.155484    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"routing-overlay\""
Nov 03 20:20:43.170985 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:43.155528    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"openshift-service-ca.crt\""
Nov 03 20:20:43.170985 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:43.155596    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-33e267\"/\"kube-root-ca.crt\""
Nov 03 20:20:43.170985 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:43.162723    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25\" (UID: \"9bfc0b88-70af-4ba3-924e-2b4f53cfdc35\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25"
Nov 03 20:20:43.170985 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:43.162803    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25\" (UID: \"9bfc0b88-70af-4ba3-924e-2b4f53cfdc35\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25"
Nov 03 20:20:43.170985 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:43.162879    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25\" (UID: \"9bfc0b88-70af-4ba3-924e-2b4f53cfdc35\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25"
Nov 03 20:20:43.175447 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:43.175408    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-944p9\" (UniqueName: \"kubernetes.io/projected/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-kube-api-access-944p9\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25\" (UID: \"9bfc0b88-70af-4ba3-924e-2b4f53cfdc35\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25"
Nov 03 20:20:43.292561 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:43.292469    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25\" (UID: \"9bfc0b88-70af-4ba3-924e-2b4f53cfdc35\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25"
Nov 03 20:20:43.292873 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:43.292849    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25\" (UID: \"9bfc0b88-70af-4ba3-924e-2b4f53cfdc35\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25"
Nov 03 20:20:43.293018 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:43.293004    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25\" (UID: \"9bfc0b88-70af-4ba3-924e-2b4f53cfdc35\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25"
Nov 03 20:20:43.293381 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:43.293359    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-944p9\" (UniqueName: \"kubernetes.io/projected/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-kube-api-access-944p9\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25\" (UID: \"9bfc0b88-70af-4ba3-924e-2b4f53cfdc35\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25"
Nov 03 20:20:43.306226 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:43.305484    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25\" (UID: \"9bfc0b88-70af-4ba3-924e-2b4f53cfdc35\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25"
Nov 03 20:20:43.307283 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:43.307254    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25\" (UID: \"9bfc0b88-70af-4ba3-924e-2b4f53cfdc35\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25"
Nov 03 20:20:43.467105 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:43.467063    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-944p9\" (UniqueName: \"kubernetes.io/projected/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-kube-api-access-944p9\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25\" (UID: \"9bfc0b88-70af-4ba3-924e-2b4f53cfdc35\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25"
Nov 03 20:20:43.678434 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:43.678333    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded"
Nov 03 20:20:43.773853 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:43.773816    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25\" (UID: \"9bfc0b88-70af-4ba3-924e-2b4f53cfdc35\") " pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25"
Nov 03 20:20:43.780231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:43.776366    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:43.809226 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:43.805057    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25"
Nov 03 20:20:43.917231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:43.917172    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 20:20:43.935379 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:43.935304    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 20:20:44.358006 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:44.351187    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 20:20:44.413116 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:44.410806    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 20:20:44.502874 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:44.502830    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 03 20:20:44.612877 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:44.612691    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:44.613546 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:44.613501    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:20:44.642196 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:44.639114    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:44.642196 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:44.639872    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 20:20:44.782858 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:44.782811    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:20:45.530851 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:45.530671    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25"]
Nov 03 20:20:45.608538 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:45.608507    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww"]
Nov 03 20:20:45.695914 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:45.695878    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25" event={"ID":"9bfc0b88-70af-4ba3-924e-2b4f53cfdc35","Type":"ContainerStarted","Data":"7949217dc7f32b3fdcdc6d130c08f44a38839b43804f5811f4b7cc42dfb270ac"}
Nov 03 20:20:45.696142 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:45.695967    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25" event={"ID":"9bfc0b88-70af-4ba3-924e-2b4f53cfdc35","Type":"ContainerStarted","Data":"64aa423b43a896ca6b9eb08b1999e5aeeb413c0f5f650aa5817fb2e2a44762f8"}
Nov 03 20:20:45.696142 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:45.696059    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25"
Nov 03 20:20:45.716661 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:45.716610    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25" podStartSLOduration=3.7165947839999998 podStartE2EDuration="3.716594784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:45.714059329 +0000 UTC m=+2594.486352359" watchObservedRunningTime="2026-11-03 20:20:45.716594784 +0000 UTC m=+2594.488887801"
Nov 03 20:20:50.609980 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:50.609926    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww" podUID="f47c6af8-7866-496c-b1ca-1ebc365884ec" containerName="payload-processing" containerID="cri-o://c5681278d737c74ddb16ff8bf740150eb4e65236722ac0e1b9489428212f6b67" gracePeriod=25
Nov 03 20:20:50.745221 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:50.745170    5591 generic.go:356] "Generic (PLEG): container finished" podID="f47c6af8-7866-496c-b1ca-1ebc365884ec" containerID="c5681278d737c74ddb16ff8bf740150eb4e65236722ac0e1b9489428212f6b67" exitCode=0
Nov 03 20:20:50.745432 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:50.745246    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww" event={"ID":"f47c6af8-7866-496c-b1ca-1ebc365884ec","Type":"ContainerDied","Data":"c5681278d737c74ddb16ff8bf740150eb4e65236722ac0e1b9489428212f6b67"}
Nov 03 20:20:50.924834 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:50.924808    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww"
Nov 03 20:20:51.036748 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:51.036710    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f47c6af8-7866-496c-b1ca-1ebc365884ec-kube-api-access-bhslf\" (UniqueName: \"kubernetes.io/projected/f47c6af8-7866-496c-b1ca-1ebc365884ec-kube-api-access-bhslf\") pod \"f47c6af8-7866-496c-b1ca-1ebc365884ec\" (UID: \"f47c6af8-7866-496c-b1ca-1ebc365884ec\") "
Nov 03 20:20:51.036973 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:51.036955    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f47c6af8-7866-496c-b1ca-1ebc365884ec-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f47c6af8-7866-496c-b1ca-1ebc365884ec-provider-credentials\") pod \"f47c6af8-7866-496c-b1ca-1ebc365884ec\" (UID: \"f47c6af8-7866-496c-b1ca-1ebc365884ec\") "
Nov 03 20:20:51.037101 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:51.037084    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f47c6af8-7866-496c-b1ca-1ebc365884ec-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f47c6af8-7866-496c-b1ca-1ebc365884ec-routing-overlay-volume\") pod \"f47c6af8-7866-496c-b1ca-1ebc365884ec\" (UID: \"f47c6af8-7866-496c-b1ca-1ebc365884ec\") "
Nov 03 20:20:51.037143 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:51.037127    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f47c6af8-7866-496c-b1ca-1ebc365884ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f47c6af8-7866-496c-b1ca-1ebc365884ec-plugins-config-volume\") pod \"f47c6af8-7866-496c-b1ca-1ebc365884ec\" (UID: \"f47c6af8-7866-496c-b1ca-1ebc365884ec\") "
Nov 03 20:20:51.037550 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:51.037492    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f47c6af8-7866-496c-b1ca-1ebc365884ec-routing-overlay-volume" pod "f47c6af8-7866-496c-b1ca-1ebc365884ec" (UID: "f47c6af8-7866-496c-b1ca-1ebc365884ec"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:51.038112 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:51.037762    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f47c6af8-7866-496c-b1ca-1ebc365884ec-plugins-config-volume" pod "f47c6af8-7866-496c-b1ca-1ebc365884ec" (UID: "f47c6af8-7866-496c-b1ca-1ebc365884ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:20:51.038112 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:51.038059    5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f47c6af8-7866-496c-b1ca-1ebc365884ec-routing-overlay-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:51.038112 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:51.038086    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f47c6af8-7866-496c-b1ca-1ebc365884ec-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:51.039750 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:51.039717    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47c6af8-7866-496c-b1ca-1ebc365884ec-provider-credentials" pod "f47c6af8-7866-496c-b1ca-1ebc365884ec" (UID: "f47c6af8-7866-496c-b1ca-1ebc365884ec"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:51.040042 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:51.039847    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47c6af8-7866-496c-b1ca-1ebc365884ec-kube-api-access-bhslf" pod "f47c6af8-7866-496c-b1ca-1ebc365884ec" (UID: "f47c6af8-7866-496c-b1ca-1ebc365884ec"). InnerVolumeSpecName "kube-api-access-bhslf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:20:51.140013 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:51.139933    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhslf\" (UniqueName: \"kubernetes.io/projected/f47c6af8-7866-496c-b1ca-1ebc365884ec-kube-api-access-bhslf\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:51.140013 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:51.139964    5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f47c6af8-7866-496c-b1ca-1ebc365884ec-provider-credentials\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:20:51.755543 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:51.755450    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww" event={"ID":"f47c6af8-7866-496c-b1ca-1ebc365884ec","Type":"ContainerDied","Data":"e6b758d70c7d7fb8aa843ce6cc275b50df70ae96b5ba850b3906859032ff7f13"}
Nov 03 20:20:51.755543 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:51.755483    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww"
Nov 03 20:20:51.755543 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:51.755519    5591 scope.go:122] "RemoveContainer" containerID="c5681278d737c74ddb16ff8bf740150eb4e65236722ac0e1b9489428212f6b67"
Nov 03 20:20:51.775699 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:51.775666    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww"]
Nov 03 20:20:51.782775 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:51.782743    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a0d4ec/payload-processing-external-model-e2e-extmod-a-a0d4ec-7d758jjww"]
Nov 03 20:20:53.356347 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:53.356310    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f47c6af8-7866-496c-b1ca-1ebc365884ec" path="/var/lib/kubelet/pods/f47c6af8-7866-496c-b1ca-1ebc365884ec/volumes"
Nov 03 20:20:54.228708 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:54.228667    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw"]
Nov 03 20:20:54.229750 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:54.229726    5591 cpu_manager.go:414] "containerMap: removing container" podUID="f47c6af8-7866-496c-b1ca-1ebc365884ec" containerName="payload-processing"
Nov 03 20:20:54.229750 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:54.229749    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f47c6af8-7866-496c-b1ca-1ebc365884ec" containerName="payload-processing"
Nov 03 20:20:54.229985 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:54.229971    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f47c6af8-7866-496c-b1ca-1ebc365884ec" containerName="payload-processing"
Nov 03 20:20:54.235651 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:54.235629    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw"
Nov 03 20:20:54.241423 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:54.241365    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0e1546\"/\"agc-cred-b09aea9ea05c\""
Nov 03 20:20:54.243508 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:54.243481    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e1546\"/\"routing-overlay\""
Nov 03 20:20:54.243643 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:54.243526    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e1546\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0e1546\""
Nov 03 20:20:54.243643 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:54.243600    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e1546\"/\"openshift-service-ca.crt\""
Nov 03 20:20:54.243643 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:54.243602    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e1546\"/\"kube-root-ca.crt\""
Nov 03 20:20:54.247918 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:54.247893    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw"]
Nov 03 20:20:54.273272 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:54.273198    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw\" (UID: \"5de06a6b-7b6b-4aba-88ce-3c50d84f403f\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw"
Nov 03 20:20:54.273272 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:54.273271    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw\" (UID: \"5de06a6b-7b6b-4aba-88ce-3c50d84f403f\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw"
Nov 03 20:20:54.273512 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:54.273343    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmzgq\" (UniqueName: \"kubernetes.io/projected/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-kube-api-access-fmzgq\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw\" (UID: \"5de06a6b-7b6b-4aba-88ce-3c50d84f403f\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw"
Nov 03 20:20:54.273512 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:54.273385    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw\" (UID: \"5de06a6b-7b6b-4aba-88ce-3c50d84f403f\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw"
Nov 03 20:20:54.374957 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:54.374921    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw\" (UID: \"5de06a6b-7b6b-4aba-88ce-3c50d84f403f\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw"
Nov 03 20:20:54.375562 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:54.375163    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw\" (UID: \"5de06a6b-7b6b-4aba-88ce-3c50d84f403f\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw"
Nov 03 20:20:54.375668 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:54.375602    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmzgq\" (UniqueName: \"kubernetes.io/projected/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-kube-api-access-fmzgq\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw\" (UID: \"5de06a6b-7b6b-4aba-88ce-3c50d84f403f\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw"
Nov 03 20:20:54.375718 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:54.375701    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw\" (UID: \"5de06a6b-7b6b-4aba-88ce-3c50d84f403f\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw"
Nov 03 20:20:54.376070 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:54.376047    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw\" (UID: \"5de06a6b-7b6b-4aba-88ce-3c50d84f403f\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw"
Nov 03 20:20:54.376561 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:54.376535    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw\" (UID: \"5de06a6b-7b6b-4aba-88ce-3c50d84f403f\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw"
Nov 03 20:20:54.407663 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:54.389503    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmzgq\" (UniqueName: \"kubernetes.io/projected/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-kube-api-access-fmzgq\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw\" (UID: \"5de06a6b-7b6b-4aba-88ce-3c50d84f403f\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw"
Nov 03 20:20:54.410530 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:54.410479    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw\" (UID: \"5de06a6b-7b6b-4aba-88ce-3c50d84f403f\") " pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw"
Nov 03 20:20:54.566239 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:54.563663    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw"
Nov 03 20:20:55.255791 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:55.255756    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw"]
Nov 03 20:20:55.261603 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:20:55.261540    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de06a6b_7b6b_4aba_88ce_3c50d84f403f.slice/crio-b535a9ae22ed701598ea1e6d3ea06e92dd1c4c0e6106f01cf1e6225809029c86 WatchSource:0}: Error finding container b535a9ae22ed701598ea1e6d3ea06e92dd1c4c0e6106f01cf1e6225809029c86: Status 404 returned error can't find the container with id b535a9ae22ed701598ea1e6d3ea06e92dd1c4c0e6106f01cf1e6225809029c86
Nov 03 20:20:55.599236 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:55.599106    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw"]
Nov 03 20:20:56.122144 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:56.122104    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw" event={"ID":"5de06a6b-7b6b-4aba-88ce-3c50d84f403f","Type":"ContainerStarted","Data":"64da85dd6c79c245bf42aa32e6a7f7a8d1002d1aca5d52f25bb1d16bd07135fc"}
Nov 03 20:20:56.122536 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:56.122177    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw" event={"ID":"5de06a6b-7b6b-4aba-88ce-3c50d84f403f","Type":"ContainerStarted","Data":"b535a9ae22ed701598ea1e6d3ea06e92dd1c4c0e6106f01cf1e6225809029c86"}
Nov 03 20:20:56.122536 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:56.122400    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw"
Nov 03 20:20:56.146186 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:56.146133    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw" podStartSLOduration=2.146118392 podStartE2EDuration="2.146118392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:56.140979474 +0000 UTC m=+2604.913272494" watchObservedRunningTime="2026-11-03 20:20:56.146118392 +0000 UTC m=+2604.918411463"
Nov 03 20:20:56.178959 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:20:56.178920    5591 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-plugins-e2e-extmod-b-0e1546: configmap "payload-processing-external-model-plugins-e2e-extmod-b-0e1546" not found
Nov 03 20:20:56.179144 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:20:56.179014    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-plugins-config-volume podName:5de06a6b-7b6b-4aba-88ce-3c50d84f403f nodeName:}" failed. No retries permitted until 2026-11-03 20:20:56.678991781 +0000 UTC m=+2605.451284795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw" (UID: "5de06a6b-7b6b-4aba-88ce-3c50d84f403f") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-0e1546" not found
Nov 03 20:20:56.179144 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:20:56.179054    5591 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0e1546/agc-cred-b09aea9ea05c: secret "agc-cred-b09aea9ea05c" not found
Nov 03 20:20:56.179144 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:20:56.179069    5591 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw: secret "agc-cred-b09aea9ea05c" not found
Nov 03 20:20:56.179144 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:20:56.179107    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-provider-credentials podName:5de06a6b-7b6b-4aba-88ce-3c50d84f403f nodeName:}" failed. No retries permitted until 2026-11-03 20:20:56.679094741 +0000 UTC m=+2605.451387738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw" (UID: "5de06a6b-7b6b-4aba-88ce-3c50d84f403f") : secret "agc-cred-b09aea9ea05c" not found
Nov 03 20:20:56.699537 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:20:56.699504    5591 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-plugins-e2e-extmod-b-0e1546: configmap "payload-processing-external-model-plugins-e2e-extmod-b-0e1546" not found
Nov 03 20:20:56.699537 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:20:56.699532    5591 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0e1546/agc-cred-b09aea9ea05c: secret "agc-cred-b09aea9ea05c" not found
Nov 03 20:20:56.700054 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:20:56.699552    5591 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw: secret "agc-cred-b09aea9ea05c" not found
Nov 03 20:20:56.700054 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:20:56.699607    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-plugins-config-volume podName:5de06a6b-7b6b-4aba-88ce-3c50d84f403f nodeName:}" failed. No retries permitted until 2026-11-03 20:20:57.699586159 +0000 UTC m=+2606.471879175 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw" (UID: "5de06a6b-7b6b-4aba-88ce-3c50d84f403f") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-0e1546" not found
Nov 03 20:20:56.700054 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:20:56.699628    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-provider-credentials podName:5de06a6b-7b6b-4aba-88ce-3c50d84f403f nodeName:}" failed. No retries permitted until 2026-11-03 20:20:57.699618638 +0000 UTC m=+2606.471911634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw" (UID: "5de06a6b-7b6b-4aba-88ce-3c50d84f403f") : secret "agc-cred-b09aea9ea05c" not found
Nov 03 20:20:56.708235 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:20:56.708194    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25"
Nov 03 20:20:57.718046 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:20:57.717966    5591 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-plugins-e2e-extmod-b-0e1546: configmap "payload-processing-external-model-plugins-e2e-extmod-b-0e1546" not found
Nov 03 20:20:57.718640 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:20:57.718057    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-plugins-config-volume podName:5de06a6b-7b6b-4aba-88ce-3c50d84f403f nodeName:}" failed. No retries permitted until 2026-11-03 20:20:59.718037507 +0000 UTC m=+2608.490330502 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw" (UID: "5de06a6b-7b6b-4aba-88ce-3c50d84f403f") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-0e1546" not found
Nov 03 20:20:57.718640 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:20:57.718559    5591 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0e1546/agc-cred-b09aea9ea05c: secret "agc-cred-b09aea9ea05c" not found
Nov 03 20:20:57.718640 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:20:57.718580    5591 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw: secret "agc-cred-b09aea9ea05c" not found
Nov 03 20:20:57.718640 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:20:57.718629    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-provider-credentials podName:5de06a6b-7b6b-4aba-88ce-3c50d84f403f nodeName:}" failed. No retries permitted until 2026-11-03 20:20:59.718613123 +0000 UTC m=+2608.490906138 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw" (UID: "5de06a6b-7b6b-4aba-88ce-3c50d84f403f") : secret "agc-cred-b09aea9ea05c" not found
Nov 03 20:20:58.731525 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:20:58.731481    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:20:59.752157 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:20:59.752122    5591 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-plugins-e2e-extmod-b-0e1546: configmap "payload-processing-external-model-plugins-e2e-extmod-b-0e1546" not found
Nov 03 20:20:59.752636 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:20:59.752191    5591 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0e1546/agc-cred-b09aea9ea05c: secret "agc-cred-b09aea9ea05c" not found
Nov 03 20:20:59.752636 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:20:59.752226    5591 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw: secret "agc-cred-b09aea9ea05c" not found
Nov 03 20:20:59.752636 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:20:59.752254    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-plugins-config-volume podName:5de06a6b-7b6b-4aba-88ce-3c50d84f403f nodeName:}" failed. No retries permitted until 2026-11-03 20:21:03.752197906 +0000 UTC m=+2612.524490919 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw" (UID: "5de06a6b-7b6b-4aba-88ce-3c50d84f403f") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-0e1546" not found
Nov 03 20:20:59.752636 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:20:59.752280    5591 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-provider-credentials podName:5de06a6b-7b6b-4aba-88ce-3c50d84f403f nodeName:}" failed. No retries permitted until 2026-11-03 20:21:03.752265634 +0000 UTC m=+2612.524558644 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw" (UID: "5de06a6b-7b6b-4aba-88ce-3c50d84f403f") : secret "agc-cred-b09aea9ea05c" not found
Nov 03 20:21:00.514272 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:00.514239    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:21:02.133228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:02.131506    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw" podUID="5de06a6b-7b6b-4aba-88ce-3c50d84f403f" containerName="payload-processing" containerID="cri-o://64da85dd6c79c245bf42aa32e6a7f7a8d1002d1aca5d52f25bb1d16bd07135fc" gracePeriod=25
Nov 03 20:21:02.616134 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:02.616100    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw"
Nov 03 20:21:02.745736 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:02.745651    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-provider-credentials\") pod \"5de06a6b-7b6b-4aba-88ce-3c50d84f403f\" (UID: \"5de06a6b-7b6b-4aba-88ce-3c50d84f403f\") "
Nov 03 20:21:02.745887 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:02.745794    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-kube-api-access-fmzgq\" (UniqueName: \"kubernetes.io/projected/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-kube-api-access-fmzgq\") pod \"5de06a6b-7b6b-4aba-88ce-3c50d84f403f\" (UID: \"5de06a6b-7b6b-4aba-88ce-3c50d84f403f\") "
Nov 03 20:21:02.745887 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:02.745872    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-routing-overlay-volume\") pod \"5de06a6b-7b6b-4aba-88ce-3c50d84f403f\" (UID: \"5de06a6b-7b6b-4aba-88ce-3c50d84f403f\") "
Nov 03 20:21:02.745979 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:02.745918    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-plugins-config-volume\") pod \"5de06a6b-7b6b-4aba-88ce-3c50d84f403f\" (UID: \"5de06a6b-7b6b-4aba-88ce-3c50d84f403f\") "
Nov 03 20:21:02.746315 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:02.746280    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-routing-overlay-volume" pod "5de06a6b-7b6b-4aba-88ce-3c50d84f403f" (UID: "5de06a6b-7b6b-4aba-88ce-3c50d84f403f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:02.746443 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:02.746380    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-plugins-config-volume" pod "5de06a6b-7b6b-4aba-88ce-3c50d84f403f" (UID: "5de06a6b-7b6b-4aba-88ce-3c50d84f403f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:21:02.746696 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:02.746681    5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-routing-overlay-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:02.746755 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:02.746697    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:02.748635 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:02.748607    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-kube-api-access-fmzgq" pod "5de06a6b-7b6b-4aba-88ce-3c50d84f403f" (UID: "5de06a6b-7b6b-4aba-88ce-3c50d84f403f"). InnerVolumeSpecName "kube-api-access-fmzgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:02.748732 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:02.748648    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-provider-credentials" pod "5de06a6b-7b6b-4aba-88ce-3c50d84f403f" (UID: "5de06a6b-7b6b-4aba-88ce-3c50d84f403f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:21:02.848831 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:02.848770    5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-provider-credentials\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:02.848831 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:02.848822    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmzgq\" (UniqueName: \"kubernetes.io/projected/5de06a6b-7b6b-4aba-88ce-3c50d84f403f-kube-api-access-fmzgq\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:21:03.197146 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:03.197111    5591 generic.go:356] "Generic (PLEG): container finished" podID="5de06a6b-7b6b-4aba-88ce-3c50d84f403f" containerID="64da85dd6c79c245bf42aa32e6a7f7a8d1002d1aca5d52f25bb1d16bd07135fc" exitCode=0
Nov 03 20:21:03.197632 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:03.197167    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw"
Nov 03 20:21:03.197632 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:03.197192    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw" event={"ID":"5de06a6b-7b6b-4aba-88ce-3c50d84f403f","Type":"ContainerDied","Data":"64da85dd6c79c245bf42aa32e6a7f7a8d1002d1aca5d52f25bb1d16bd07135fc"}
Nov 03 20:21:03.197632 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:03.197300    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw" event={"ID":"5de06a6b-7b6b-4aba-88ce-3c50d84f403f","Type":"ContainerDied","Data":"b535a9ae22ed701598ea1e6d3ea06e92dd1c4c0e6106f01cf1e6225809029c86"}
Nov 03 20:21:03.197632 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:03.197340    5591 scope.go:122] "RemoveContainer" containerID="64da85dd6c79c245bf42aa32e6a7f7a8d1002d1aca5d52f25bb1d16bd07135fc"
Nov 03 20:21:03.211861 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:03.211836    5591 scope.go:122] "RemoveContainer" containerID="64da85dd6c79c245bf42aa32e6a7f7a8d1002d1aca5d52f25bb1d16bd07135fc"
Nov 03 20:21:03.212243 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:21:03.212216    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64da85dd6c79c245bf42aa32e6a7f7a8d1002d1aca5d52f25bb1d16bd07135fc\": container with ID starting with 64da85dd6c79c245bf42aa32e6a7f7a8d1002d1aca5d52f25bb1d16bd07135fc not found: ID does not exist" containerID="64da85dd6c79c245bf42aa32e6a7f7a8d1002d1aca5d52f25bb1d16bd07135fc"
Nov 03 20:21:03.212333 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:03.212253    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64da85dd6c79c245bf42aa32e6a7f7a8d1002d1aca5d52f25bb1d16bd07135fc"} err="failed to get container status \"64da85dd6c79c245bf42aa32e6a7f7a8d1002d1aca5d52f25bb1d16bd07135fc\": rpc error: code = NotFound desc = could not find container \"64da85dd6c79c245bf42aa32e6a7f7a8d1002d1aca5d52f25bb1d16bd07135fc\": container with ID starting with 64da85dd6c79c245bf42aa32e6a7f7a8d1002d1aca5d52f25bb1d16bd07135fc not found: ID does not exist"
Nov 03 20:21:03.229096 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:03.229055    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw"]
Nov 03 20:21:03.274250 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:03.269714    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0e1546/payload-processing-external-model-e2e-extmod-b-0e1546-57f6gd7zw"]
Nov 03 20:21:03.354828 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:03.354796    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5de06a6b-7b6b-4aba-88ce-3c50d84f403f" path="/var/lib/kubelet/pods/5de06a6b-7b6b-4aba-88ce-3c50d84f403f/volumes"
Nov 03 20:21:05.038949 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:05.038912    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb"]
Nov 03 20:21:05.040277 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:05.040252    5591 cpu_manager.go:414] "containerMap: removing container" podUID="5de06a6b-7b6b-4aba-88ce-3c50d84f403f" containerName="payload-processing"
Nov 03 20:21:05.040277 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:05.040276    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5de06a6b-7b6b-4aba-88ce-3c50d84f403f" containerName="payload-processing"
Nov 03 20:21:05.040506 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:05.040493    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="5de06a6b-7b6b-4aba-88ce-3c50d84f403f" containerName="payload-processing"
Nov 03 20:21:05.045860 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:05.045839    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb"
Nov 03 20:21:05.051329 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:05.051307    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-51f23d5569d8\""
Nov 03 20:21:05.052323 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:05.051625    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 20:21:05.052504 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:05.052133    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 20:21:05.052504 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:05.052144    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 20:21:05.052504 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:05.052169    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 20:21:05.052728 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:05.052273    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-5lpr4\""
Nov 03 20:21:05.054281 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:05.054258    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb"]
Nov 03 20:21:05.144482 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:05.144448    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c8b12594-22bc-4f44-999d-4fcc5628ad5a-routing-overlay-volume\") pod \"payload-processing-external-model-86b5bf4bf-2jszb\" (UID: \"c8b12594-22bc-4f44-999d-4fcc5628ad5a\") " pod="models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb"
Nov 03 20:21:05.144654 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:05.144635    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8b12594-22bc-4f44-999d-4fcc5628ad5a-plugins-config-volume\") pod \"payload-processing-external-model-86b5bf4bf-2jszb\" (UID: \"c8b12594-22bc-4f44-999d-4fcc5628ad5a\") " pod="models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb"
Nov 03 20:21:05.144715 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:05.144702    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbbfx\" (UniqueName: \"kubernetes.io/projected/c8b12594-22bc-4f44-999d-4fcc5628ad5a-kube-api-access-pbbfx\") pod \"payload-processing-external-model-86b5bf4bf-2jszb\" (UID: \"c8b12594-22bc-4f44-999d-4fcc5628ad5a\") " pod="models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb"
Nov 03 20:21:05.144836 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:05.144823    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c8b12594-22bc-4f44-999d-4fcc5628ad5a-provider-credentials\") pod \"payload-processing-external-model-86b5bf4bf-2jszb\" (UID: \"c8b12594-22bc-4f44-999d-4fcc5628ad5a\") " pod="models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb"
Nov 03 20:21:05.246974 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:05.246930    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c8b12594-22bc-4f44-999d-4fcc5628ad5a-provider-credentials\") pod \"payload-processing-external-model-86b5bf4bf-2jszb\" (UID: \"c8b12594-22bc-4f44-999d-4fcc5628ad5a\") " pod="models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb"
Nov 03 20:21:05.247187 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:05.247143    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c8b12594-22bc-4f44-999d-4fcc5628ad5a-routing-overlay-volume\") pod \"payload-processing-external-model-86b5bf4bf-2jszb\" (UID: \"c8b12594-22bc-4f44-999d-4fcc5628ad5a\") " pod="models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb"
Nov 03 20:21:05.247433 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:05.247409    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8b12594-22bc-4f44-999d-4fcc5628ad5a-plugins-config-volume\") pod \"payload-processing-external-model-86b5bf4bf-2jszb\" (UID: \"c8b12594-22bc-4f44-999d-4fcc5628ad5a\") " pod="models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb"
Nov 03 20:21:05.247502 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:05.247468    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbbfx\" (UniqueName: \"kubernetes.io/projected/c8b12594-22bc-4f44-999d-4fcc5628ad5a-kube-api-access-pbbfx\") pod \"payload-processing-external-model-86b5bf4bf-2jszb\" (UID: \"c8b12594-22bc-4f44-999d-4fcc5628ad5a\") " pod="models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb"
Nov 03 20:21:05.249135 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:05.249102    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c8b12594-22bc-4f44-999d-4fcc5628ad5a-routing-overlay-volume\") pod \"payload-processing-external-model-86b5bf4bf-2jszb\" (UID: \"c8b12594-22bc-4f44-999d-4fcc5628ad5a\") " pod="models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb"
Nov 03 20:21:05.250165 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:05.250141    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8b12594-22bc-4f44-999d-4fcc5628ad5a-plugins-config-volume\") pod \"payload-processing-external-model-86b5bf4bf-2jszb\" (UID: \"c8b12594-22bc-4f44-999d-4fcc5628ad5a\") " pod="models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb"
Nov 03 20:21:05.262885 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:05.262854    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c8b12594-22bc-4f44-999d-4fcc5628ad5a-provider-credentials\") pod \"payload-processing-external-model-86b5bf4bf-2jszb\" (UID: \"c8b12594-22bc-4f44-999d-4fcc5628ad5a\") " pod="models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb"
Nov 03 20:21:05.265853 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:05.265826    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbbfx\" (UniqueName: \"kubernetes.io/projected/c8b12594-22bc-4f44-999d-4fcc5628ad5a-kube-api-access-pbbfx\") pod \"payload-processing-external-model-86b5bf4bf-2jszb\" (UID: \"c8b12594-22bc-4f44-999d-4fcc5628ad5a\") " pod="models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb"
Nov 03 20:21:05.364237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:05.364090    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb"
Nov 03 20:21:05.861104 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:05.860631    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb"]
Nov 03 20:21:05.887968 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:21:05.887938    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b12594_22bc_4f44_999d_4fcc5628ad5a.slice/crio-5905794cc5bd604781af5c99e31411ccdfecc58a347e808c1d1a148af557540e WatchSource:0}: Error finding container 5905794cc5bd604781af5c99e31411ccdfecc58a347e808c1d1a148af557540e: Status 404 returned error can't find the container with id 5905794cc5bd604781af5c99e31411ccdfecc58a347e808c1d1a148af557540e
Nov 03 20:21:06.469718 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:06.469676    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb" event={"ID":"c8b12594-22bc-4f44-999d-4fcc5628ad5a","Type":"ContainerStarted","Data":"74dd73142fc423f9bd234fec98a6e650a0ea3187016bf86fb2083f68d69df430"}
Nov 03 20:21:06.477013 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:06.476469    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb" event={"ID":"c8b12594-22bc-4f44-999d-4fcc5628ad5a","Type":"ContainerStarted","Data":"5905794cc5bd604781af5c99e31411ccdfecc58a347e808c1d1a148af557540e"}
Nov 03 20:21:06.478863 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:06.478834    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb"
Nov 03 20:21:06.495557 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:06.495508    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb" podStartSLOduration=1.495481231 podStartE2EDuration="1.495481231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:06.493150554 +0000 UTC m=+2615.265443582" watchObservedRunningTime="2026-11-03 20:21:06.495481231 +0000 UTC m=+2615.267774252"
Nov 03 20:21:10.113543 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:10.113499    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q"]
Nov 03 20:21:10.120072 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:10.120042    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q"
Nov 03 20:21:10.126632 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:10.126599    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e7b70d-dockercfg-jd4cp\""
Nov 03 20:21:10.133903 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:10.133874    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q"]
Nov 03 20:21:10.234290 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:10.217519    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e7b70d\""
Nov 03 20:21:10.260746 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:10.260676    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q\" (UID: \"ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q"
Nov 03 20:21:10.261282 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:10.261256    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shgw8\" (UniqueName: \"kubernetes.io/projected/ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd-kube-api-access-shgw8\") pod \"payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q\" (UID: \"ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q"
Nov 03 20:21:10.406306 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:10.402334    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q\" (UID: \"ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q"
Nov 03 20:21:10.406306 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:10.402695    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shgw8\" (UniqueName: \"kubernetes.io/projected/ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd-kube-api-access-shgw8\") pod \"payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q\" (UID: \"ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q"
Nov 03 20:21:10.406306 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:10.404746    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q\" (UID: \"ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q"
Nov 03 20:21:10.490226 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:10.484455    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shgw8\" (UniqueName: \"kubernetes.io/projected/ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd-kube-api-access-shgw8\") pod \"payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q\" (UID: \"ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q"
Nov 03 20:21:10.764809 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:10.764766    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q"
Nov 03 20:21:11.068462 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:11.068429    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q"]
Nov 03 20:21:11.069261 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:21:11.069230    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded8d7bd5_8fb7_4f6e_b986_9fa1480d8bfd.slice/crio-8a863a1da93806a121afbcab033d9774d39b3c8a9f42b05994f13374b235164b WatchSource:0}: Error finding container 8a863a1da93806a121afbcab033d9774d39b3c8a9f42b05994f13374b235164b: Status 404 returned error can't find the container with id 8a863a1da93806a121afbcab033d9774d39b3c8a9f42b05994f13374b235164b
Nov 03 20:21:11.573516 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:11.573476    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q" event={"ID":"ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd","Type":"ContainerStarted","Data":"7b7caf3dc4d00c8e109a035729a4b41d04308b9c25d078047390001bc8d5add6"}
Nov 03 20:21:11.573936 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:11.573543    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q" event={"ID":"ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd","Type":"ContainerStarted","Data":"8a863a1da93806a121afbcab033d9774d39b3c8a9f42b05994f13374b235164b"}
Nov 03 20:21:11.573936 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:11.573574    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q"
Nov 03 20:21:11.596663 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:11.596604    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q" podStartSLOduration=1.5965910810000001 podStartE2EDuration="1.596591081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:11.591872924 +0000 UTC m=+2620.364165939" watchObservedRunningTime="2026-11-03 20:21:11.596591081 +0000 UTC m=+2620.368884093"
Nov 03 20:21:18.545986 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:18.545952    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb"
Nov 03 20:21:22.589495 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:22.589415    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q"
Nov 03 20:21:28.753263 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:21:28.753171    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:21:30.419369 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:30.419309    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 20:21:30.463370 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:30.463322    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:30.523354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:30.523305    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:30.586052 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:30.585993    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:30.830517 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:30.830453    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:31.433238 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:31.432346    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 20:21:31.442725 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:31.442681    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 20:21:31.466961 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:31.464698    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:31.466961 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:31.465312    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 20:21:31.466961 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:31.466638    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:31.499989 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:31.499960    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 20:21:31.524363 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:31.524319    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:31.525354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:31.525322    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded"
Nov 03 20:21:31.525594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:31.525574    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 20:21:31.545544 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:31.545492    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:31.587820 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:31.587778    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:21:31.589043 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:31.589018    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 20:21:31.795254 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:31.795185    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 20:21:31.832596 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:31.832093    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 20:21:31.885731 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:21:31.885702    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 20:21:58.726911 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:21:58.726873    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:22:06.822291 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:06.821342    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:22:14.796649 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:14.796616    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"]
Nov 03 20:22:14.797173 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:14.796877    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p" podUID="1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b" containerName="istio-proxy" containerID="cri-o://007c05e37eb70000c378f832736aceb97542b9853f342da2754550f23438f9e1" gracePeriod=30
Nov 03 20:22:20.291581 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.291551    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:22:20.329752 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.329711    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk"]
Nov 03 20:22:20.330894 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.330867    5591 cpu_manager.go:414] "containerMap: removing container" podUID="1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b" containerName="istio-proxy"
Nov 03 20:22:20.330894 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.330888    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b" containerName="istio-proxy"
Nov 03 20:22:20.331105 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.331080    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b" containerName="istio-proxy"
Nov 03 20:22:20.335312 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.335283    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk"
Nov 03 20:22:20.349097 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.349055    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk"]
Nov 03 20:22:20.349474 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.349450    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-74dfc6-serving-cert\""
Nov 03 20:22:20.454134 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.454043    5591 generic.go:356] "Generic (PLEG): container finished" podID="1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b" containerID="007c05e37eb70000c378f832736aceb97542b9853f342da2754550f23438f9e1" exitCode=0
Nov 03 20:22:20.454134 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.454110    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"
Nov 03 20:22:20.454134 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.454124    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p" event={"ID":"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b","Type":"ContainerDied","Data":"007c05e37eb70000c378f832736aceb97542b9853f342da2754550f23438f9e1"}
Nov 03 20:22:20.454432 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.454236    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p" event={"ID":"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b","Type":"ContainerDied","Data":"ec9ac6c08980f43f5b1bf95ad776c0c308cab9b489a88fd7eb7a45f8cc29e04a"}
Nov 03 20:22:20.454432 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.454261    5591 scope.go:122] "RemoveContainer" containerID="007c05e37eb70000c378f832736aceb97542b9853f342da2754550f23438f9e1"
Nov 03 20:22:20.458627 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.458606    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-kube-api-access-gkz9q\" (UniqueName: \"kubernetes.io/projected/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-kube-api-access-gkz9q\") pod \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") "
Nov 03 20:22:20.458776 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.458759    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-data\") pod \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") "
Nov 03 20:22:20.458853 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.458841    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-podinfo\") pod \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") "
Nov 03 20:22:20.458917 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.458903    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-envoy\") pod \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") "
Nov 03 20:22:20.459005 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.458983    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-workload-socket\") pod \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") "
Nov 03 20:22:20.459132 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.459028    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-credential-socket\") pod \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") "
Nov 03 20:22:20.459132 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.459092    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-token\" (UniqueName: \"kubernetes.io/projected/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-token\") pod \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") "
Nov 03 20:22:20.459288 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.459148    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istiod-ca-cert\") pod \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") "
Nov 03 20:22:20.459288 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.459195    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-workload-certs\") pod \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\" (UID: \"1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b\") "
Nov 03 20:22:20.459288 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.459260    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-workload-socket" pod "1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b" (UID: "1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:20.459577 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.459559    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istiod-ca-cert" pod "1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b" (UID: "1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:22:20.459728 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.459685    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-credential-socket" pod "1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b" (UID: "1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:20.460053 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.459827    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a28774-81b4-4051-9e4f-cb8fbe703942-openshift-service-ca\") pod \"maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk\" (UID: \"07a28774-81b4-4051-9e4f-cb8fbe703942\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk"
Nov 03 20:22:20.460053 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.459929    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-workload-certs" pod "1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b" (UID: "1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:20.460053 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.460004    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-data" pod "1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b" (UID: "1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:20.460365 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.460340    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a28774-81b4-4051-9e4f-cb8fbe703942-metrics-certs\") pod \"maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk\" (UID: \"07a28774-81b4-4051-9e4f-cb8fbe703942\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk"
Nov 03 20:22:20.460478 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.460461    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07a28774-81b4-4051-9e4f-cb8fbe703942-maas-api-tls\") pod \"maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk\" (UID: \"07a28774-81b4-4051-9e4f-cb8fbe703942\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk"
Nov 03 20:22:20.460546 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.460495    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfw44\" (UniqueName: \"kubernetes.io/projected/07a28774-81b4-4051-9e4f-cb8fbe703942-kube-api-access-jfw44\") pod \"maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk\" (UID: \"07a28774-81b4-4051-9e4f-cb8fbe703942\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk"
Nov 03 20:22:20.460739 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.460712    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:20.460824 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.460752    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:20.460824 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.460772    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:20.460824 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.460787    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:20.460824 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.460800    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:20.461617 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.461594    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-podinfo" pod "1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b" (UID: "1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:22:20.461757 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.461626    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-kube-api-access-gkz9q" pod "1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b" (UID: "1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b"). InnerVolumeSpecName "kube-api-access-gkz9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:20.461865 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.461804    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-envoy" pod "1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b" (UID: "1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:22:20.462138 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.462110    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-token" pod "1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b" (UID: "1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:20.477502 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.477476    5591 scope.go:122] "RemoveContainer" containerID="007c05e37eb70000c378f832736aceb97542b9853f342da2754550f23438f9e1"
Nov 03 20:22:20.477920 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:22:20.477898    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"007c05e37eb70000c378f832736aceb97542b9853f342da2754550f23438f9e1\": container with ID starting with 007c05e37eb70000c378f832736aceb97542b9853f342da2754550f23438f9e1 not found: ID does not exist" containerID="007c05e37eb70000c378f832736aceb97542b9853f342da2754550f23438f9e1"
Nov 03 20:22:20.478052 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.477931    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"007c05e37eb70000c378f832736aceb97542b9853f342da2754550f23438f9e1"} err="failed to get container status \"007c05e37eb70000c378f832736aceb97542b9853f342da2754550f23438f9e1\": rpc error: code = NotFound desc = could not find container \"007c05e37eb70000c378f832736aceb97542b9853f342da2754550f23438f9e1\": container with ID starting with 007c05e37eb70000c378f832736aceb97542b9853f342da2754550f23438f9e1 not found: ID does not exist"
Nov 03 20:22:20.562149 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.562114    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07a28774-81b4-4051-9e4f-cb8fbe703942-maas-api-tls\") pod \"maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk\" (UID: \"07a28774-81b4-4051-9e4f-cb8fbe703942\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk"
Nov 03 20:22:20.562345 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.562161    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfw44\" (UniqueName: \"kubernetes.io/projected/07a28774-81b4-4051-9e4f-cb8fbe703942-kube-api-access-jfw44\") pod \"maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk\" (UID: \"07a28774-81b4-4051-9e4f-cb8fbe703942\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk"
Nov 03 20:22:20.562345 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.562246    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a28774-81b4-4051-9e4f-cb8fbe703942-openshift-service-ca\") pod \"maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk\" (UID: \"07a28774-81b4-4051-9e4f-cb8fbe703942\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk"
Nov 03 20:22:20.562511 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.562498    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a28774-81b4-4051-9e4f-cb8fbe703942-metrics-certs\") pod \"maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk\" (UID: \"07a28774-81b4-4051-9e4f-cb8fbe703942\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk"
Nov 03 20:22:20.562599 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.562588    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:20.562650 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.562602    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:20.562650 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.562611    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:20.562650 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.562620    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkz9q\" (UniqueName: \"kubernetes.io/projected/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b-kube-api-access-gkz9q\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:20.562979 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.562957    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a28774-81b4-4051-9e4f-cb8fbe703942-openshift-service-ca\") pod \"maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk\" (UID: \"07a28774-81b4-4051-9e4f-cb8fbe703942\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk"
Nov 03 20:22:20.564937 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.564894    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07a28774-81b4-4051-9e4f-cb8fbe703942-maas-api-tls\") pod \"maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk\" (UID: \"07a28774-81b4-4051-9e4f-cb8fbe703942\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk"
Nov 03 20:22:20.565132 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.565110    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a28774-81b4-4051-9e4f-cb8fbe703942-metrics-certs\") pod \"maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk\" (UID: \"07a28774-81b4-4051-9e4f-cb8fbe703942\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk"
Nov 03 20:22:20.593536 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.593499    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfw44\" (UniqueName: \"kubernetes.io/projected/07a28774-81b4-4051-9e4f-cb8fbe703942-kube-api-access-jfw44\") pod \"maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk\" (UID: \"07a28774-81b4-4051-9e4f-cb8fbe703942\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk"
Nov 03 20:22:20.685997 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.685961    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk"
Nov 03 20:22:20.789293 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.789266    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"]
Nov 03 20:22:20.794811 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.794743    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-trt6p"]
Nov 03 20:22:20.976953 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.976923    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk"]
Nov 03 20:22:20.977430 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:22:20.977405    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a28774_81b4_4051_9e4f_cb8fbe703942.slice/crio-f03a9fd2319a73a70d6f8267bb50b7fa5b292207d7b8c66b53dfff2623f109f9 WatchSource:0}: Error finding container f03a9fd2319a73a70d6f8267bb50b7fa5b292207d7b8c66b53dfff2623f109f9: Status 404 returned error can't find the container with id f03a9fd2319a73a70d6f8267bb50b7fa5b292207d7b8c66b53dfff2623f109f9
Nov 03 20:22:20.978988 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:20.978969    5591 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:22:21.354184 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:21.354152    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b" path="/var/lib/kubelet/pods/1a3c3c2e-65cc-4468-b7bb-3f65107b2e9b/volumes"
Nov 03 20:22:21.468859 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:21.468820    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk" event={"ID":"07a28774-81b4-4051-9e4f-cb8fbe703942","Type":"ContainerStarted","Data":"f03a9fd2319a73a70d6f8267bb50b7fa5b292207d7b8c66b53dfff2623f109f9"}
Nov 03 20:22:22.479804 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:22.479761    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk" event={"ID":"07a28774-81b4-4051-9e4f-cb8fbe703942","Type":"ContainerStarted","Data":"59a40d58c3df89999f733881b03c925eb23f1bcbeecc41e8aa0cf9bdb6d3103b"}
Nov 03 20:22:22.480312 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:22.479853    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk"
Nov 03 20:22:22.501239 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:22.501083    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk" podStartSLOduration=4.038935962 podStartE2EDuration="4.501065383s" totalImagesPullingTime="462.129421ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:22.500897696 +0000 UTC m=+2691.273190713" watchObservedRunningTime="2026-11-03 20:22:22.501065383 +0000 UTC m=+2691.273358402"
Nov 03 20:22:28.520186 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:28.520149    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk"
Nov 03 20:22:28.728106 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:22:28.728066    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:22:31.569773 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:31.568425    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:31.811983 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:31.807325    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:31.819195 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:31.817372    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:31.854731 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:31.854625    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:31.898687 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:31.893501    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:22:33.849161 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:33.849129    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 20:22:33.851438 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:33.851413    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 20:22:45.763557 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:45.762529    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"]
Nov 03 20:22:45.773625 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:45.771501    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"
Nov 03 20:22:45.789604 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:45.787086    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"]
Nov 03 20:22:45.793460 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:45.793436    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e7b70d\"/\"openshift-service-ca.crt\""
Nov 03 20:22:45.793820 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:45.793480    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e7b70d\"/\"kube-root-ca.crt\""
Nov 03 20:22:45.809283 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:45.809186    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e7b70d\"/\"default-dockercfg-v4txc\""
Nov 03 20:22:45.809495 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:45.809265    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e7b70d\"/\"test-model-e7b70d-kserve-self-signed-certs\""
Nov 03 20:22:45.816562 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:45.816527    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-dshm\") pod \"test-model-e7b70d-kserve-677dbc94c7-zk4kl\" (UID: \"97472d8f-eb02-4974-9a3d-275a65ca555b\") " pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"
Nov 03 20:22:45.816947 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:45.816921    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-home\") pod \"test-model-e7b70d-kserve-677dbc94c7-zk4kl\" (UID: \"97472d8f-eb02-4974-9a3d-275a65ca555b\") " pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"
Nov 03 20:22:45.817170 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:45.817147    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-tmp-dir\") pod \"test-model-e7b70d-kserve-677dbc94c7-zk4kl\" (UID: \"97472d8f-eb02-4974-9a3d-275a65ca555b\") " pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"
Nov 03 20:22:45.817400 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:45.817381    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8pnq\" (UniqueName: \"kubernetes.io/projected/97472d8f-eb02-4974-9a3d-275a65ca555b-kube-api-access-t8pnq\") pod \"test-model-e7b70d-kserve-677dbc94c7-zk4kl\" (UID: \"97472d8f-eb02-4974-9a3d-275a65ca555b\") " pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"
Nov 03 20:22:45.817557 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:45.817533    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97472d8f-eb02-4974-9a3d-275a65ca555b-tls-certs\") pod \"test-model-e7b70d-kserve-677dbc94c7-zk4kl\" (UID: \"97472d8f-eb02-4974-9a3d-275a65ca555b\") " pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"
Nov 03 20:22:45.817696 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:45.817614    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-model-cache\") pod \"test-model-e7b70d-kserve-677dbc94c7-zk4kl\" (UID: \"97472d8f-eb02-4974-9a3d-275a65ca555b\") " pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"
Nov 03 20:22:45.921261 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:45.919967    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-dshm\") pod \"test-model-e7b70d-kserve-677dbc94c7-zk4kl\" (UID: \"97472d8f-eb02-4974-9a3d-275a65ca555b\") " pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"
Nov 03 20:22:45.921261 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:45.920077    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-home\") pod \"test-model-e7b70d-kserve-677dbc94c7-zk4kl\" (UID: \"97472d8f-eb02-4974-9a3d-275a65ca555b\") " pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"
Nov 03 20:22:45.921261 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:45.920140    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-tmp-dir\") pod \"test-model-e7b70d-kserve-677dbc94c7-zk4kl\" (UID: \"97472d8f-eb02-4974-9a3d-275a65ca555b\") " pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"
Nov 03 20:22:45.921261 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:45.920217    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8pnq\" (UniqueName: \"kubernetes.io/projected/97472d8f-eb02-4974-9a3d-275a65ca555b-kube-api-access-t8pnq\") pod \"test-model-e7b70d-kserve-677dbc94c7-zk4kl\" (UID: \"97472d8f-eb02-4974-9a3d-275a65ca555b\") " pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"
Nov 03 20:22:45.921261 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:45.920244    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97472d8f-eb02-4974-9a3d-275a65ca555b-tls-certs\") pod \"test-model-e7b70d-kserve-677dbc94c7-zk4kl\" (UID: \"97472d8f-eb02-4974-9a3d-275a65ca555b\") " pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"
Nov 03 20:22:45.921261 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:45.920296    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-model-cache\") pod \"test-model-e7b70d-kserve-677dbc94c7-zk4kl\" (UID: \"97472d8f-eb02-4974-9a3d-275a65ca555b\") " pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"
Nov 03 20:22:45.921261 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:45.920716    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-model-cache\") pod \"test-model-e7b70d-kserve-677dbc94c7-zk4kl\" (UID: \"97472d8f-eb02-4974-9a3d-275a65ca555b\") " pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"
Nov 03 20:22:45.921261 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:45.920863    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-tmp-dir\") pod \"test-model-e7b70d-kserve-677dbc94c7-zk4kl\" (UID: \"97472d8f-eb02-4974-9a3d-275a65ca555b\") " pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"
Nov 03 20:22:45.921261 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:45.921126    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-home\") pod \"test-model-e7b70d-kserve-677dbc94c7-zk4kl\" (UID: \"97472d8f-eb02-4974-9a3d-275a65ca555b\") " pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"
Nov 03 20:22:45.923739 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:45.923711    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-dshm\") pod \"test-model-e7b70d-kserve-677dbc94c7-zk4kl\" (UID: \"97472d8f-eb02-4974-9a3d-275a65ca555b\") " pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"
Nov 03 20:22:45.924961 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:45.924939    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97472d8f-eb02-4974-9a3d-275a65ca555b-tls-certs\") pod \"test-model-e7b70d-kserve-677dbc94c7-zk4kl\" (UID: \"97472d8f-eb02-4974-9a3d-275a65ca555b\") " pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"
Nov 03 20:22:45.936092 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:45.936060    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8pnq\" (UniqueName: \"kubernetes.io/projected/97472d8f-eb02-4974-9a3d-275a65ca555b-kube-api-access-t8pnq\") pod \"test-model-e7b70d-kserve-677dbc94c7-zk4kl\" (UID: \"97472d8f-eb02-4974-9a3d-275a65ca555b\") " pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"
Nov 03 20:22:46.117281 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:46.116548    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"
Nov 03 20:22:46.446251 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:46.446217    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"]
Nov 03 20:22:46.447888 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:22:46.447863    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97472d8f_eb02_4974_9a3d_275a65ca555b.slice/crio-ace566ecd9b1e3a6eade0ee5726fca71a4b08f0bbff444af43f5da5d9596df7a WatchSource:0}: Error finding container ace566ecd9b1e3a6eade0ee5726fca71a4b08f0bbff444af43f5da5d9596df7a: Status 404 returned error can't find the container with id ace566ecd9b1e3a6eade0ee5726fca71a4b08f0bbff444af43f5da5d9596df7a
Nov 03 20:22:47.164132 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:47.164100    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl" event={"ID":"97472d8f-eb02-4974-9a3d-275a65ca555b","Type":"ContainerStarted","Data":"ea308f815a7c96ece2e89e4415f6eafbf41e1edb6cd91755aaf8fad8651086e8"}
Nov 03 20:22:47.164558 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:47.164188    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl" event={"ID":"97472d8f-eb02-4974-9a3d-275a65ca555b","Type":"ContainerStarted","Data":"ace566ecd9b1e3a6eade0ee5726fca71a4b08f0bbff444af43f5da5d9596df7a"}
Nov 03 20:22:47.190803 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:47.190742    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl" podStartSLOduration=2.190723751 podStartE2EDuration="2.190723751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:47.185618642 +0000 UTC m=+2715.957911676" watchObservedRunningTime="2026-11-03 20:22:47.190723751 +0000 UTC m=+2715.963016771"
Nov 03 20:22:48.117400 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:48.117365    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"
Nov 03 20:22:50.829278 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:50.829234    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb"]
Nov 03 20:22:55.830417 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:55.830378    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb" podUID="c8b12594-22bc-4f44-999d-4fcc5628ad5a" containerName="payload-processing" containerID="cri-o://74dd73142fc423f9bd234fec98a6e650a0ea3187016bf86fb2083f68d69df430" gracePeriod=25
Nov 03 20:22:56.117051 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.117022    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"
Nov 03 20:22:56.134817 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.134790    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"
Nov 03 20:22:56.149277 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.149254    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb"
Nov 03 20:22:56.186506 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.186467    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf"]
Nov 03 20:22:56.187519 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.187492    5591 cpu_manager.go:414] "containerMap: removing container" podUID="c8b12594-22bc-4f44-999d-4fcc5628ad5a" containerName="payload-processing"
Nov 03 20:22:56.187519 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.187513    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b12594-22bc-4f44-999d-4fcc5628ad5a" containerName="payload-processing"
Nov 03 20:22:56.187792 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.187777    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b12594-22bc-4f44-999d-4fcc5628ad5a" containerName="payload-processing"
Nov 03 20:22:56.192221 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.192186    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf"
Nov 03 20:22:56.197167 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.197135    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"kube-root-ca.crt\""
Nov 03 20:22:56.202721 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.202696    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf"]
Nov 03 20:22:56.202858 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.202837    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"routing-overlay\""
Nov 03 20:22:56.202958 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.202941    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8b7351\""
Nov 03 20:22:56.203036 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.203016    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"openshift-service-ca.crt\""
Nov 03 20:22:56.220559 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.220528    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8b12594-22bc-4f44-999d-4fcc5628ad5a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c8b12594-22bc-4f44-999d-4fcc5628ad5a-routing-overlay-volume\") pod \"c8b12594-22bc-4f44-999d-4fcc5628ad5a\" (UID: \"c8b12594-22bc-4f44-999d-4fcc5628ad5a\") "
Nov 03 20:22:56.220746 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.220603    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b12594-22bc-4f44-999d-4fcc5628ad5a-kube-api-access-pbbfx\" (UniqueName: \"kubernetes.io/projected/c8b12594-22bc-4f44-999d-4fcc5628ad5a-kube-api-access-pbbfx\") pod \"c8b12594-22bc-4f44-999d-4fcc5628ad5a\" (UID: \"c8b12594-22bc-4f44-999d-4fcc5628ad5a\") "
Nov 03 20:22:56.220746 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.220730    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b12594-22bc-4f44-999d-4fcc5628ad5a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c8b12594-22bc-4f44-999d-4fcc5628ad5a-provider-credentials\") pod \"c8b12594-22bc-4f44-999d-4fcc5628ad5a\" (UID: \"c8b12594-22bc-4f44-999d-4fcc5628ad5a\") "
Nov 03 20:22:56.220875 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.220817    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8b12594-22bc-4f44-999d-4fcc5628ad5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8b12594-22bc-4f44-999d-4fcc5628ad5a-plugins-config-volume\") pod \"c8b12594-22bc-4f44-999d-4fcc5628ad5a\" (UID: \"c8b12594-22bc-4f44-999d-4fcc5628ad5a\") "
Nov 03 20:22:56.221094 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.221072    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b9f9140-739a-4f76-9165-6af845847971-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf\" (UID: \"1b9f9140-739a-4f76-9165-6af845847971\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf"
Nov 03 20:22:56.221375 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.221334    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b9f9140-739a-4f76-9165-6af845847971-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf\" (UID: \"1b9f9140-739a-4f76-9165-6af845847971\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf"
Nov 03 20:22:56.221623 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.221597    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9f9140-739a-4f76-9165-6af845847971-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf\" (UID: \"1b9f9140-739a-4f76-9165-6af845847971\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf"
Nov 03 20:22:56.221990 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.221640    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b12594-22bc-4f44-999d-4fcc5628ad5a-routing-overlay-volume" pod "c8b12594-22bc-4f44-999d-4fcc5628ad5a" (UID: "c8b12594-22bc-4f44-999d-4fcc5628ad5a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:22:56.221990 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.221662    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b12594-22bc-4f44-999d-4fcc5628ad5a-plugins-config-volume" pod "c8b12594-22bc-4f44-999d-4fcc5628ad5a" (UID: "c8b12594-22bc-4f44-999d-4fcc5628ad5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:22:56.221990 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.221693    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxs82\" (UniqueName: \"kubernetes.io/projected/1b9f9140-739a-4f76-9165-6af845847971-kube-api-access-cxs82\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf\" (UID: \"1b9f9140-739a-4f76-9165-6af845847971\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf"
Nov 03 20:22:56.222268 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.222191    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8b12594-22bc-4f44-999d-4fcc5628ad5a-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:56.222268 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.222238    5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c8b12594-22bc-4f44-999d-4fcc5628ad5a-routing-overlay-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:56.223629 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.223607    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b12594-22bc-4f44-999d-4fcc5628ad5a-kube-api-access-pbbfx" pod "c8b12594-22bc-4f44-999d-4fcc5628ad5a" (UID: "c8b12594-22bc-4f44-999d-4fcc5628ad5a"). InnerVolumeSpecName "kube-api-access-pbbfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:56.223728 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.223646    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b12594-22bc-4f44-999d-4fcc5628ad5a-provider-credentials" pod "c8b12594-22bc-4f44-999d-4fcc5628ad5a" (UID: "c8b12594-22bc-4f44-999d-4fcc5628ad5a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:22:56.256860 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.256831    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"agc-cred-c5d06cfd93a8\""
Nov 03 20:22:56.257029 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.256972    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8b7351\"/\"payload-processing-external-model-e2e--3d0fafa7-dockercfg-c5kt8\""
Nov 03 20:22:56.289955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.289919    5591 generic.go:356] "Generic (PLEG): container finished" podID="c8b12594-22bc-4f44-999d-4fcc5628ad5a" containerID="74dd73142fc423f9bd234fec98a6e650a0ea3187016bf86fb2083f68d69df430" exitCode=0
Nov 03 20:22:56.290115 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.289964    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb"
Nov 03 20:22:56.290115 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.289999    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb" event={"ID":"c8b12594-22bc-4f44-999d-4fcc5628ad5a","Type":"ContainerDied","Data":"74dd73142fc423f9bd234fec98a6e650a0ea3187016bf86fb2083f68d69df430"}
Nov 03 20:22:56.290115 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.290057    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb" event={"ID":"c8b12594-22bc-4f44-999d-4fcc5628ad5a","Type":"ContainerDied","Data":"5905794cc5bd604781af5c99e31411ccdfecc58a347e808c1d1a148af557540e"}
Nov 03 20:22:56.290115 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.290088    5591 scope.go:122] "RemoveContainer" containerID="74dd73142fc423f9bd234fec98a6e650a0ea3187016bf86fb2083f68d69df430"
Nov 03 20:22:56.303823 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.303800    5591 scope.go:122] "RemoveContainer" containerID="74dd73142fc423f9bd234fec98a6e650a0ea3187016bf86fb2083f68d69df430"
Nov 03 20:22:56.304082 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:22:56.304063    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74dd73142fc423f9bd234fec98a6e650a0ea3187016bf86fb2083f68d69df430\": container with ID starting with 74dd73142fc423f9bd234fec98a6e650a0ea3187016bf86fb2083f68d69df430 not found: ID does not exist" containerID="74dd73142fc423f9bd234fec98a6e650a0ea3187016bf86fb2083f68d69df430"
Nov 03 20:22:56.304138 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.304093    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74dd73142fc423f9bd234fec98a6e650a0ea3187016bf86fb2083f68d69df430"} err="failed to get container status \"74dd73142fc423f9bd234fec98a6e650a0ea3187016bf86fb2083f68d69df430\": rpc error: code = NotFound desc = could not find container \"74dd73142fc423f9bd234fec98a6e650a0ea3187016bf86fb2083f68d69df430\": container with ID starting with 74dd73142fc423f9bd234fec98a6e650a0ea3187016bf86fb2083f68d69df430 not found: ID does not exist"
Nov 03 20:22:56.305438 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.305413    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"
Nov 03 20:22:56.319246 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.319183    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb"]
Nov 03 20:22:56.323601 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.323571    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b9f9140-739a-4f76-9165-6af845847971-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf\" (UID: \"1b9f9140-739a-4f76-9165-6af845847971\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf"
Nov 03 20:22:56.323733 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.323713    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9f9140-739a-4f76-9165-6af845847971-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf\" (UID: \"1b9f9140-739a-4f76-9165-6af845847971\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf"
Nov 03 20:22:56.323778 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.323749    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-86b5bf4bf-2jszb"]
Nov 03 20:22:56.323815 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.323750    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxs82\" (UniqueName: \"kubernetes.io/projected/1b9f9140-739a-4f76-9165-6af845847971-kube-api-access-cxs82\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf\" (UID: \"1b9f9140-739a-4f76-9165-6af845847971\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf"
Nov 03 20:22:56.323853 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.323825    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b9f9140-739a-4f76-9165-6af845847971-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf\" (UID: \"1b9f9140-739a-4f76-9165-6af845847971\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf"
Nov 03 20:22:56.324109 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.324091    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbbfx\" (UniqueName: \"kubernetes.io/projected/c8b12594-22bc-4f44-999d-4fcc5628ad5a-kube-api-access-pbbfx\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:56.324197 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.324117    5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c8b12594-22bc-4f44-999d-4fcc5628ad5a-provider-credentials\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:22:56.324473 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.324452    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b9f9140-739a-4f76-9165-6af845847971-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf\" (UID: \"1b9f9140-739a-4f76-9165-6af845847971\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf"
Nov 03 20:22:56.324573 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.324552    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9f9140-739a-4f76-9165-6af845847971-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf\" (UID: \"1b9f9140-739a-4f76-9165-6af845847971\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf"
Nov 03 20:22:56.326698 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.326680    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b9f9140-739a-4f76-9165-6af845847971-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf\" (UID: \"1b9f9140-739a-4f76-9165-6af845847971\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf"
Nov 03 20:22:56.335936 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.335915    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxs82\" (UniqueName: \"kubernetes.io/projected/1b9f9140-739a-4f76-9165-6af845847971-kube-api-access-cxs82\") pod \"payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf\" (UID: \"1b9f9140-739a-4f76-9165-6af845847971\") " pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf"
Nov 03 20:22:56.519762 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.519730    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf"
Nov 03 20:22:56.822139 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:22:56.822109    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b9f9140_739a_4f76_9165_6af845847971.slice/crio-6a642878dc6307e427c0addb91899aedf86ce0134ffe9f3e8f4ab2b38b06d5d5 WatchSource:0}: Error finding container 6a642878dc6307e427c0addb91899aedf86ce0134ffe9f3e8f4ab2b38b06d5d5: Status 404 returned error can't find the container with id 6a642878dc6307e427c0addb91899aedf86ce0134ffe9f3e8f4ab2b38b06d5d5
Nov 03 20:22:56.823744 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.823720    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf"]
Nov 03 20:22:56.913655 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.913624    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"]
Nov 03 20:22:56.919959 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.919823    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"
Nov 03 20:22:56.927861 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.927828    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"]
Nov 03 20:22:56.930495 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.930469    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-74dfc6\"/\"default-dockercfg-smmsw\""
Nov 03 20:22:56.931948 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.931899    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-74dfc6\"/\"openshift-service-ca.crt\""
Nov 03 20:22:56.932066 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.932034    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-74dfc6\"/\"kube-root-ca.crt\""
Nov 03 20:22:56.936256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.936193    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmrld\" (UniqueName: \"kubernetes.io/projected/666c0175-cb77-4af6-8793-81a7a684c1f2-kube-api-access-mmrld\") pod \"test-model-74dfc6-kserve-85d9d78f47-zjm87\" (UID: \"666c0175-cb77-4af6-8793-81a7a684c1f2\") " pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"
Nov 03 20:22:56.936465 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.936303    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-dshm\") pod \"test-model-74dfc6-kserve-85d9d78f47-zjm87\" (UID: \"666c0175-cb77-4af6-8793-81a7a684c1f2\") " pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"
Nov 03 20:22:56.936539 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.936514    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-home\") pod \"test-model-74dfc6-kserve-85d9d78f47-zjm87\" (UID: \"666c0175-cb77-4af6-8793-81a7a684c1f2\") " pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"
Nov 03 20:22:56.936605 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.936553    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/666c0175-cb77-4af6-8793-81a7a684c1f2-tls-certs\") pod \"test-model-74dfc6-kserve-85d9d78f47-zjm87\" (UID: \"666c0175-cb77-4af6-8793-81a7a684c1f2\") " pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"
Nov 03 20:22:56.936727 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.936660    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-tmp-dir\") pod \"test-model-74dfc6-kserve-85d9d78f47-zjm87\" (UID: \"666c0175-cb77-4af6-8793-81a7a684c1f2\") " pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"
Nov 03 20:22:56.936848 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.936826    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-model-cache\") pod \"test-model-74dfc6-kserve-85d9d78f47-zjm87\" (UID: \"666c0175-cb77-4af6-8793-81a7a684c1f2\") " pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"
Nov 03 20:22:56.969738 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:56.969696    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-74dfc6\"/\"test-model-74dfc6-kserve-self-signed-certs\""
Nov 03 20:22:57.038616 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:57.038572    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-tmp-dir\") pod \"test-model-74dfc6-kserve-85d9d78f47-zjm87\" (UID: \"666c0175-cb77-4af6-8793-81a7a684c1f2\") " pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"
Nov 03 20:22:57.038828 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:57.038722    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-model-cache\") pod \"test-model-74dfc6-kserve-85d9d78f47-zjm87\" (UID: \"666c0175-cb77-4af6-8793-81a7a684c1f2\") " pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"
Nov 03 20:22:57.038909 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:57.038867    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmrld\" (UniqueName: \"kubernetes.io/projected/666c0175-cb77-4af6-8793-81a7a684c1f2-kube-api-access-mmrld\") pod \"test-model-74dfc6-kserve-85d9d78f47-zjm87\" (UID: \"666c0175-cb77-4af6-8793-81a7a684c1f2\") " pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"
Nov 03 20:22:57.038974 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:57.038936    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-dshm\") pod \"test-model-74dfc6-kserve-85d9d78f47-zjm87\" (UID: \"666c0175-cb77-4af6-8793-81a7a684c1f2\") " pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"
Nov 03 20:22:57.039026 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:57.038992    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-tmp-dir\") pod \"test-model-74dfc6-kserve-85d9d78f47-zjm87\" (UID: \"666c0175-cb77-4af6-8793-81a7a684c1f2\") " pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"
Nov 03 20:22:57.039066 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:57.039041    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-model-cache\") pod \"test-model-74dfc6-kserve-85d9d78f47-zjm87\" (UID: \"666c0175-cb77-4af6-8793-81a7a684c1f2\") " pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"
Nov 03 20:22:57.039135 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:57.039117    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-home\") pod \"test-model-74dfc6-kserve-85d9d78f47-zjm87\" (UID: \"666c0175-cb77-4af6-8793-81a7a684c1f2\") " pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"
Nov 03 20:22:57.039187 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:57.039170    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/666c0175-cb77-4af6-8793-81a7a684c1f2-tls-certs\") pod \"test-model-74dfc6-kserve-85d9d78f47-zjm87\" (UID: \"666c0175-cb77-4af6-8793-81a7a684c1f2\") " pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"
Nov 03 20:22:57.039444 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:57.039424    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-home\") pod \"test-model-74dfc6-kserve-85d9d78f47-zjm87\" (UID: \"666c0175-cb77-4af6-8793-81a7a684c1f2\") " pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"
Nov 03 20:22:57.041669 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:57.041635    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-dshm\") pod \"test-model-74dfc6-kserve-85d9d78f47-zjm87\" (UID: \"666c0175-cb77-4af6-8793-81a7a684c1f2\") " pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"
Nov 03 20:22:57.041981 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:57.041965    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/666c0175-cb77-4af6-8793-81a7a684c1f2-tls-certs\") pod \"test-model-74dfc6-kserve-85d9d78f47-zjm87\" (UID: \"666c0175-cb77-4af6-8793-81a7a684c1f2\") " pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"
Nov 03 20:22:57.051352 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:57.051314    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmrld\" (UniqueName: \"kubernetes.io/projected/666c0175-cb77-4af6-8793-81a7a684c1f2-kube-api-access-mmrld\") pod \"test-model-74dfc6-kserve-85d9d78f47-zjm87\" (UID: \"666c0175-cb77-4af6-8793-81a7a684c1f2\") " pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"
Nov 03 20:22:57.244340 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:57.242516    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"
Nov 03 20:22:57.305336 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:57.305133    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf" event={"ID":"1b9f9140-739a-4f76-9165-6af845847971","Type":"ContainerStarted","Data":"a16e5529c29277947c6f685c5b2e0ea1300314b9dfabeeb54f6ec4fc19c8724c"}
Nov 03 20:22:57.305758 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:57.305416    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf" event={"ID":"1b9f9140-739a-4f76-9165-6af845847971","Type":"ContainerStarted","Data":"6a642878dc6307e427c0addb91899aedf86ce0134ffe9f3e8f4ab2b38b06d5d5"}
Nov 03 20:22:57.306336 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:57.306284    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf"
Nov 03 20:22:57.344030 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:57.343972    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf" podStartSLOduration=3.343950578 podStartE2EDuration="3.343950578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:57.326686402 +0000 UTC m=+2726.098979411" watchObservedRunningTime="2026-11-03 20:22:57.343950578 +0000 UTC m=+2726.116243600"
Nov 03 20:22:57.367226 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:57.362904    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8b12594-22bc-4f44-999d-4fcc5628ad5a" path="/var/lib/kubelet/pods/c8b12594-22bc-4f44-999d-4fcc5628ad5a/volumes"
Nov 03 20:22:57.616560 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:57.616530    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"]
Nov 03 20:22:57.616750 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:22:57.616727    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod666c0175_cb77_4af6_8793_81a7a684c1f2.slice/crio-bf145c0ce1af8f48c7d94eb7255ee6e4dd2052d42b9a2bfb2edc6e47a143aa12 WatchSource:0}: Error finding container bf145c0ce1af8f48c7d94eb7255ee6e4dd2052d42b9a2bfb2edc6e47a143aa12: Status 404 returned error can't find the container with id bf145c0ce1af8f48c7d94eb7255ee6e4dd2052d42b9a2bfb2edc6e47a143aa12
Nov 03 20:22:58.317016 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:58.316974    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87" event={"ID":"666c0175-cb77-4af6-8793-81a7a684c1f2","Type":"ContainerStarted","Data":"bab068229c1d465d063b7c4099715f510fbf984434b03a23faaacdd2a5634e0b"}
Nov 03 20:22:58.317016 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:58.317023    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87" event={"ID":"666c0175-cb77-4af6-8793-81a7a684c1f2","Type":"ContainerStarted","Data":"bf145c0ce1af8f48c7d94eb7255ee6e4dd2052d42b9a2bfb2edc6e47a143aa12"}
Nov 03 20:22:58.331878 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:58.331842    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:22:58.336094 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:58.336046    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87" podStartSLOduration=2.3360322829999998 podStartE2EDuration="2.336032283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:58.334765951 +0000 UTC m=+2727.107058970" watchObservedRunningTime="2026-11-03 20:22:58.336032283 +0000 UTC m=+2727.108325301"
Nov 03 20:22:58.751656 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:22:58.751531    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:22:59.243101 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:22:59.243062    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"
Nov 03 20:23:07.243313 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:23:07.243230    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"
Nov 03 20:23:07.259940 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:23:07.259911    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"
Nov 03 20:23:07.419119 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:23:07.419088    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"
Nov 03 20:23:08.321383 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:23:08.321350    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf"
Nov 03 20:23:28.727586 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:23:28.727535    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:23:36.818155 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:23:36.818121    5591 scope.go:122] "RemoveContainer" containerID="15cb524a90c8ccab44ebfc36fbad0c3e55d33602392fdd29412909f8591a2860"
Nov 03 20:23:58.727946 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:23:58.727906    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:24:00.889130 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:24:00.889092    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:24:28.726936 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:24:28.726894    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:24:41.243732 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:24:41.243695    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:24:58.966910 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:24:58.966736    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:25:08.653301 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:08.653262    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-7ec3de/e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx"]
Nov 03 20:25:08.653897 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:08.653490    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-7ec3de/e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx" podUID="d7710d70-6f64-455a-93de-8ef183cede56" containerName="sim" containerID="cri-o://2bc2389cb49d89d35a7d9ac6677f95704cf1628e0351ec8b6cf3d5a36771a77a" gracePeriod=30
Nov 03 20:25:09.128797 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:09.128597    5591 generic.go:356] "Generic (PLEG): container finished" podID="d7710d70-6f64-455a-93de-8ef183cede56" containerID="2bc2389cb49d89d35a7d9ac6677f95704cf1628e0351ec8b6cf3d5a36771a77a" exitCode=0
Nov 03 20:25:09.128797 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:09.128719    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-7ec3de/e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx" event={"ID":"d7710d70-6f64-455a-93de-8ef183cede56","Type":"ContainerDied","Data":"2bc2389cb49d89d35a7d9ac6677f95704cf1628e0351ec8b6cf3d5a36771a77a"}
Nov 03 20:25:09.540869 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:09.540842    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-7ec3de/e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx"
Nov 03 20:25:09.677060 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:09.677018    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7710d70-6f64-455a-93de-8ef183cede56-tls\" (UniqueName: \"kubernetes.io/secret/d7710d70-6f64-455a-93de-8ef183cede56-tls\") pod \"d7710d70-6f64-455a-93de-8ef183cede56\" (UID: \"d7710d70-6f64-455a-93de-8ef183cede56\") "
Nov 03 20:25:09.677765 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:09.677144    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7710d70-6f64-455a-93de-8ef183cede56-kube-api-access-9frmk\" (UniqueName: \"kubernetes.io/projected/d7710d70-6f64-455a-93de-8ef183cede56-kube-api-access-9frmk\") pod \"d7710d70-6f64-455a-93de-8ef183cede56\" (UID: \"d7710d70-6f64-455a-93de-8ef183cede56\") "
Nov 03 20:25:09.691433 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:09.691393    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7710d70-6f64-455a-93de-8ef183cede56-kube-api-access-9frmk" pod "d7710d70-6f64-455a-93de-8ef183cede56" (UID: "d7710d70-6f64-455a-93de-8ef183cede56"). InnerVolumeSpecName "kube-api-access-9frmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:09.717105 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:09.717026    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7710d70-6f64-455a-93de-8ef183cede56-tls" pod "d7710d70-6f64-455a-93de-8ef183cede56" (UID: "d7710d70-6f64-455a-93de-8ef183cede56"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:25:09.779415 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:09.779382    5591 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d7710d70-6f64-455a-93de-8ef183cede56-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:09.779415 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:09.779414    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9frmk\" (UniqueName: \"kubernetes.io/projected/d7710d70-6f64-455a-93de-8ef183cede56-kube-api-access-9frmk\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:10.138553 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:10.138527    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-7ec3de/e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx"
Nov 03 20:25:10.138553 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:10.138559    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-7ec3de/e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx" event={"ID":"d7710d70-6f64-455a-93de-8ef183cede56","Type":"ContainerDied","Data":"ac7e98d8386c7abc6b8e4288b7e499cf6939fc08858c6ddbfe8e3bcb998ef48c"}
Nov 03 20:25:10.138831 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:10.138608    5591 scope.go:122] "RemoveContainer" containerID="2bc2389cb49d89d35a7d9ac6677f95704cf1628e0351ec8b6cf3d5a36771a77a"
Nov 03 20:25:10.164280 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:10.164252    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-7ec3de/e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx"]
Nov 03 20:25:10.169572 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:10.169549    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-7ec3de/e2e-usage-sim-5a5376-6d9fb9b4f6-xkwgx"]
Nov 03 20:25:11.354874 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:11.354841    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7710d70-6f64-455a-93de-8ef183cede56" path="/var/lib/kubelet/pods/d7710d70-6f64-455a-93de-8ef183cede56/volumes"
Nov 03 20:25:12.115836 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:12.115675    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25"]
Nov 03 20:25:17.116732 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:17.116669    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25" podUID="9bfc0b88-70af-4ba3-924e-2b4f53cfdc35" containerName="payload-processing" containerID="cri-o://7949217dc7f32b3fdcdc6d130c08f44a38839b43804f5811f4b7cc42dfb270ac" gracePeriod=25
Nov 03 20:25:17.580675 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:17.580645    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25"
Nov 03 20:25:17.637878 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:17.637831    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-plugins-config-volume\") pod \"9bfc0b88-70af-4ba3-924e-2b4f53cfdc35\" (UID: \"9bfc0b88-70af-4ba3-924e-2b4f53cfdc35\") "
Nov 03 20:25:17.638069 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:17.638038    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-kube-api-access-944p9\" (UniqueName: \"kubernetes.io/projected/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-kube-api-access-944p9\") pod \"9bfc0b88-70af-4ba3-924e-2b4f53cfdc35\" (UID: \"9bfc0b88-70af-4ba3-924e-2b4f53cfdc35\") "
Nov 03 20:25:17.638302 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:17.638276    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-plugins-config-volume" pod "9bfc0b88-70af-4ba3-924e-2b4f53cfdc35" (UID: "9bfc0b88-70af-4ba3-924e-2b4f53cfdc35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:17.638355 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:17.638314    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-routing-overlay-volume\") pod \"9bfc0b88-70af-4ba3-924e-2b4f53cfdc35\" (UID: \"9bfc0b88-70af-4ba3-924e-2b4f53cfdc35\") "
Nov 03 20:25:17.638405 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:17.638379    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-provider-credentials\") pod \"9bfc0b88-70af-4ba3-924e-2b4f53cfdc35\" (UID: \"9bfc0b88-70af-4ba3-924e-2b4f53cfdc35\") "
Nov 03 20:25:17.638696 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:17.638676    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-routing-overlay-volume" pod "9bfc0b88-70af-4ba3-924e-2b4f53cfdc35" (UID: "9bfc0b88-70af-4ba3-924e-2b4f53cfdc35"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:25:17.639314 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:17.639289    5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-routing-overlay-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:17.639417 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:17.639346    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:17.640789 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:17.640763    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-kube-api-access-944p9" pod "9bfc0b88-70af-4ba3-924e-2b4f53cfdc35" (UID: "9bfc0b88-70af-4ba3-924e-2b4f53cfdc35"). InnerVolumeSpecName "kube-api-access-944p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:17.640878 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:17.640813    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-provider-credentials" pod "9bfc0b88-70af-4ba3-924e-2b4f53cfdc35" (UID: "9bfc0b88-70af-4ba3-924e-2b4f53cfdc35"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:25:17.741594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:17.741547    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-944p9\" (UniqueName: \"kubernetes.io/projected/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-kube-api-access-944p9\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:17.741594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:17.741592    5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35-provider-credentials\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:25:18.286672 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:18.286634    5591 generic.go:356] "Generic (PLEG): container finished" podID="9bfc0b88-70af-4ba3-924e-2b4f53cfdc35" containerID="7949217dc7f32b3fdcdc6d130c08f44a38839b43804f5811f4b7cc42dfb270ac" exitCode=0
Nov 03 20:25:18.287191 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:18.286689    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25"
Nov 03 20:25:18.287191 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:18.286723    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25" event={"ID":"9bfc0b88-70af-4ba3-924e-2b4f53cfdc35","Type":"ContainerDied","Data":"7949217dc7f32b3fdcdc6d130c08f44a38839b43804f5811f4b7cc42dfb270ac"}
Nov 03 20:25:18.287191 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:18.286849    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25" event={"ID":"9bfc0b88-70af-4ba3-924e-2b4f53cfdc35","Type":"ContainerDied","Data":"64aa423b43a896ca6b9eb08b1999e5aeeb413c0f5f650aa5817fb2e2a44762f8"}
Nov 03 20:25:18.287191 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:18.286882    5591 scope.go:122] "RemoveContainer" containerID="7949217dc7f32b3fdcdc6d130c08f44a38839b43804f5811f4b7cc42dfb270ac"
Nov 03 20:25:18.304058 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:18.304034    5591 scope.go:122] "RemoveContainer" containerID="7949217dc7f32b3fdcdc6d130c08f44a38839b43804f5811f4b7cc42dfb270ac"
Nov 03 20:25:18.304486 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:25:18.304459    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7949217dc7f32b3fdcdc6d130c08f44a38839b43804f5811f4b7cc42dfb270ac\": container with ID starting with 7949217dc7f32b3fdcdc6d130c08f44a38839b43804f5811f4b7cc42dfb270ac not found: ID does not exist" containerID="7949217dc7f32b3fdcdc6d130c08f44a38839b43804f5811f4b7cc42dfb270ac"
Nov 03 20:25:18.304587 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:18.304498    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7949217dc7f32b3fdcdc6d130c08f44a38839b43804f5811f4b7cc42dfb270ac"} err="failed to get container status \"7949217dc7f32b3fdcdc6d130c08f44a38839b43804f5811f4b7cc42dfb270ac\": rpc error: code = NotFound desc = could not find container \"7949217dc7f32b3fdcdc6d130c08f44a38839b43804f5811f4b7cc42dfb270ac\": container with ID starting with 7949217dc7f32b3fdcdc6d130c08f44a38839b43804f5811f4b7cc42dfb270ac not found: ID does not exist"
Nov 03 20:25:18.315710 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:18.315682    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25"]
Nov 03 20:25:18.320330 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:18.320302    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-33e267/payload-processing-external-model-e2e-extmod-a-33e267-5b68bsp25"]
Nov 03 20:25:19.358516 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:19.358416    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bfc0b88-70af-4ba3-924e-2b4f53cfdc35" path="/var/lib/kubelet/pods/9bfc0b88-70af-4ba3-924e-2b4f53cfdc35/volumes"
Nov 03 20:25:22.105088 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:22.105047    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45"]
Nov 03 20:25:22.106290 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:22.106267    5591 cpu_manager.go:414] "containerMap: removing container" podUID="d7710d70-6f64-455a-93de-8ef183cede56" containerName="sim"
Nov 03 20:25:22.106290 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:22.106290    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7710d70-6f64-455a-93de-8ef183cede56" containerName="sim"
Nov 03 20:25:22.106467 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:22.106347    5591 cpu_manager.go:414] "containerMap: removing container" podUID="9bfc0b88-70af-4ba3-924e-2b4f53cfdc35" containerName="payload-processing"
Nov 03 20:25:22.106467 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:22.106356    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bfc0b88-70af-4ba3-924e-2b4f53cfdc35" containerName="payload-processing"
Nov 03 20:25:22.106605 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:22.106589    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7710d70-6f64-455a-93de-8ef183cede56" containerName="sim"
Nov 03 20:25:22.106669 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:22.106623    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bfc0b88-70af-4ba3-924e-2b4f53cfdc35" containerName="payload-processing"
Nov 03 20:25:22.112329 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:22.112304    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45"
Nov 03 20:25:22.118787 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:22.118758    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d2aa3b\""
Nov 03 20:25:22.125254 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:22.125185    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45"]
Nov 03 20:25:22.183393 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:22.183350    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blvqr\" (UniqueName: \"kubernetes.io/projected/5e92bb59-2a3f-4286-90eb-d33f8f857c9d-kube-api-access-blvqr\") pod \"payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45\" (UID: \"5e92bb59-2a3f-4286-90eb-d33f8f857c9d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45"
Nov 03 20:25:22.183613 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:22.183430    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e92bb59-2a3f-4286-90eb-d33f8f857c9d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45\" (UID: \"5e92bb59-2a3f-4286-90eb-d33f8f857c9d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45"
Nov 03 20:25:22.318229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:22.317410    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blvqr\" (UniqueName: \"kubernetes.io/projected/5e92bb59-2a3f-4286-90eb-d33f8f857c9d-kube-api-access-blvqr\") pod \"payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45\" (UID: \"5e92bb59-2a3f-4286-90eb-d33f8f857c9d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45"
Nov 03 20:25:22.318229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:22.317498    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e92bb59-2a3f-4286-90eb-d33f8f857c9d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45\" (UID: \"5e92bb59-2a3f-4286-90eb-d33f8f857c9d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45"
Nov 03 20:25:22.323229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:22.318583    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e92bb59-2a3f-4286-90eb-d33f8f857c9d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45\" (UID: \"5e92bb59-2a3f-4286-90eb-d33f8f857c9d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45"
Nov 03 20:25:22.369041 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:22.368960    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blvqr\" (UniqueName: \"kubernetes.io/projected/5e92bb59-2a3f-4286-90eb-d33f8f857c9d-kube-api-access-blvqr\") pod \"payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45\" (UID: \"5e92bb59-2a3f-4286-90eb-d33f8f857c9d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45"
Nov 03 20:25:22.439606 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:22.439568    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45"
Nov 03 20:25:22.848920 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:22.848879    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45"]
Nov 03 20:25:23.498281 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:23.491720    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45" event={"ID":"5e92bb59-2a3f-4286-90eb-d33f8f857c9d","Type":"ContainerStarted","Data":"68f2ba4ae81a9a43aa31ffbaf6a1a571f81d264ef36971f7b22c9fcde2bb727e"}
Nov 03 20:25:23.498281 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:23.491812    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45"
Nov 03 20:25:23.498281 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:23.491837    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45" event={"ID":"5e92bb59-2a3f-4286-90eb-d33f8f857c9d","Type":"ContainerStarted","Data":"81b9f79e56fb25678648008876cf5680c5eff2ea76b70fd19c475017fd6bfe7a"}
Nov 03 20:25:28.725225 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:25:28.725164    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:25:34.505155 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:34.505122    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45"
Nov 03 20:25:34.528355 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:34.528306    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45" podStartSLOduration=12.5282935 podStartE2EDuration="12.5282935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:23.533559417 +0000 UTC m=+2872.305852436" watchObservedRunningTime="2026-11-03 20:25:34.5282935 +0000 UTC m=+2883.300586519"
Nov 03 20:25:42.902876 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:25:42.902795    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:25:58.726253 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:25:58.726193    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:26:19.614745 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:19.614713    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"]
Nov 03 20:26:19.615264 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:19.614917    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv" podUID="7e760329-c9c8-4d93-8efb-8aaa938899b5" containerName="istio-proxy" containerID="cri-o://7723978d5daa9201a3078a7e34e842e0b4c794ef1610927634de33ff2c465e3e" gracePeriod=30
Nov 03 20:26:23.316843 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:23.316804    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"]
Nov 03 20:26:23.317449 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:23.317030    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr" podUID="b043886c-68e1-48d8-9ec4-8ecd4601a753" containerName="istio-proxy" containerID="cri-o://7caaeec4c314cef19e22813fb950cf5661b3f97fe0c26e48513fc643739a1b62" gracePeriod=30
Nov 03 20:26:24.964968 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:24.964937    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:26:25.002712 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.002677    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"]
Nov 03 20:26:25.003769 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.003750    5591 cpu_manager.go:414] "containerMap: removing container" podUID="7e760329-c9c8-4d93-8efb-8aaa938899b5" containerName="istio-proxy"
Nov 03 20:26:25.003839 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.003771    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e760329-c9c8-4d93-8efb-8aaa938899b5" containerName="istio-proxy"
Nov 03 20:26:25.003938 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.003928    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e760329-c9c8-4d93-8efb-8aaa938899b5" containerName="istio-proxy"
Nov 03 20:26:25.008093 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.008074    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.023347 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.023302    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-759b04-openshift-default-dockercfg-jr6b2\""
Nov 03 20:26:25.023905 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.023871    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-token\" (UniqueName: \"kubernetes.io/projected/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-token\") pod \"7e760329-c9c8-4d93-8efb-8aaa938899b5\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") "
Nov 03 20:26:25.024075 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.023974    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-envoy\") pod \"7e760329-c9c8-4d93-8efb-8aaa938899b5\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") "
Nov 03 20:26:25.024701 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.024110    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-podinfo\") pod \"7e760329-c9c8-4d93-8efb-8aaa938899b5\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") "
Nov 03 20:26:25.024862 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.024768    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-credential-socket\") pod \"7e760329-c9c8-4d93-8efb-8aaa938899b5\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") "
Nov 03 20:26:25.025501 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.025044    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-credential-socket" pod "7e760329-c9c8-4d93-8efb-8aaa938899b5" (UID: "7e760329-c9c8-4d93-8efb-8aaa938899b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:25.025501 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.025193    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-data\") pod \"7e760329-c9c8-4d93-8efb-8aaa938899b5\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") "
Nov 03 20:26:25.025501 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.025305    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-workload-socket\") pod \"7e760329-c9c8-4d93-8efb-8aaa938899b5\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") "
Nov 03 20:26:25.025501 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.025369    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e760329-c9c8-4d93-8efb-8aaa938899b5-kube-api-access-5p89x\" (UniqueName: \"kubernetes.io/projected/7e760329-c9c8-4d93-8efb-8aaa938899b5-kube-api-access-5p89x\") pod \"7e760329-c9c8-4d93-8efb-8aaa938899b5\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") "
Nov 03 20:26:25.025501 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.025439    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e760329-c9c8-4d93-8efb-8aaa938899b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e760329-c9c8-4d93-8efb-8aaa938899b5-istiod-ca-cert\") pod \"7e760329-c9c8-4d93-8efb-8aaa938899b5\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") "
Nov 03 20:26:25.025923 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.025571    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-workload-certs\") pod \"7e760329-c9c8-4d93-8efb-8aaa938899b5\" (UID: \"7e760329-c9c8-4d93-8efb-8aaa938899b5\") "
Nov 03 20:26:25.026053 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.026015    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c55ca26-6f34-4632-bf90-ab132d63fe78-istiod-ca-cert\") pod \"e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") " pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.026347 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.026329    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-token\") pod \"e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") " pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.026574 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.026557    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-workload-socket\") pod \"e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") " pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.026800 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.026779    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-envoy\") pod \"e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") " pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.027005 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.026971    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-podinfo\") pod \"e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") " pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.027287 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.027252    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-workload-certs\") pod \"e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") " pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.028188 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.027809    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-workload-certs" pod "7e760329-c9c8-4d93-8efb-8aaa938899b5" (UID: "7e760329-c9c8-4d93-8efb-8aaa938899b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:25.028188 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.027825    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-data" pod "7e760329-c9c8-4d93-8efb-8aaa938899b5" (UID: "7e760329-c9c8-4d93-8efb-8aaa938899b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:25.028188 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.028115    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-workload-socket" pod "7e760329-c9c8-4d93-8efb-8aaa938899b5" (UID: "7e760329-c9c8-4d93-8efb-8aaa938899b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:25.028188 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.028157    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e760329-c9c8-4d93-8efb-8aaa938899b5-istiod-ca-cert" pod "7e760329-c9c8-4d93-8efb-8aaa938899b5" (UID: "7e760329-c9c8-4d93-8efb-8aaa938899b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:25.028509 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.027531    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dlng\" (UniqueName: \"kubernetes.io/projected/6c55ca26-6f34-4632-bf90-ab132d63fe78-kube-api-access-6dlng\") pod \"e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") " pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.028808 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.028789    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-data\") pod \"e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") " pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.029005 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.028971    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-credential-socket\") pod \"e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") " pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.029440 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.029408    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:25.029595 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.029581    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:25.029723 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.029710    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:25.029847 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.029833    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e760329-c9c8-4d93-8efb-8aaa938899b5-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:25.029969 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.029957    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:25.031481 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.031459    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e760329-c9c8-4d93-8efb-8aaa938899b5-kube-api-access-5p89x" pod "7e760329-c9c8-4d93-8efb-8aaa938899b5" (UID: "7e760329-c9c8-4d93-8efb-8aaa938899b5"). InnerVolumeSpecName "kube-api-access-5p89x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:25.034351 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.034298    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-podinfo" pod "7e760329-c9c8-4d93-8efb-8aaa938899b5" (UID: "7e760329-c9c8-4d93-8efb-8aaa938899b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:25.043704 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.043665    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-token" pod "7e760329-c9c8-4d93-8efb-8aaa938899b5" (UID: "7e760329-c9c8-4d93-8efb-8aaa938899b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:25.044246 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.044188    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-envoy" pod "7e760329-c9c8-4d93-8efb-8aaa938899b5" (UID: "7e760329-c9c8-4d93-8efb-8aaa938899b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:25.046570 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.046491    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"]
Nov 03 20:26:25.131375 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.131341    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-data\") pod \"e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") " pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.131565 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.131387    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-credential-socket\") pod \"e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") " pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.131565 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.131492    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c55ca26-6f34-4632-bf90-ab132d63fe78-istiod-ca-cert\") pod \"e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") " pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.131565 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.131539    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-token\") pod \"e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") " pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.131726 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.131589    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-workload-socket\") pod \"e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") " pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.131726 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.131631    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-envoy\") pod \"e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") " pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.131726 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.131662    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-podinfo\") pod \"e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") " pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.131726 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.131720    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-workload-certs\") pod \"e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") " pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.131943 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.131806    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dlng\" (UniqueName: \"kubernetes.io/projected/6c55ca26-6f34-4632-bf90-ab132d63fe78-kube-api-access-6dlng\") pod \"e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") " pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.131943 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.131878    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-data\") pod \"e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") " pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.131943 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.131896    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-credential-socket\") pod \"e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") " pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.132139 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.131983    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-workload-socket\") pod \"e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") " pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.132139 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.132052    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:25.132139 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.132074    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:25.132139 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.132091    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e760329-c9c8-4d93-8efb-8aaa938899b5-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:25.132139 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.132108    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p89x\" (UniqueName: \"kubernetes.io/projected/7e760329-c9c8-4d93-8efb-8aaa938899b5-kube-api-access-5p89x\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:25.132428 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.132230    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-workload-certs\") pod \"e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") " pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.132428 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.132318    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c55ca26-6f34-4632-bf90-ab132d63fe78-istiod-ca-cert\") pod \"e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") " pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.134286 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.134262    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-envoy\") pod \"e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") " pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.134436 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.134419    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-podinfo\") pod \"e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") " pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.144314 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.144291    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-token\") pod \"e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") " pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.144407 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.144337    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dlng\" (UniqueName: \"kubernetes.io/projected/6c55ca26-6f34-4632-bf90-ab132d63fe78-kube-api-access-6dlng\") pod \"e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") " pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.187108 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.187033    5591 generic.go:356] "Generic (PLEG): container finished" podID="7e760329-c9c8-4d93-8efb-8aaa938899b5" containerID="7723978d5daa9201a3078a7e34e842e0b4c794ef1610927634de33ff2c465e3e" exitCode=0
Nov 03 20:26:25.187108 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.187098    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"
Nov 03 20:26:25.187319 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.187118    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv" event={"ID":"7e760329-c9c8-4d93-8efb-8aaa938899b5","Type":"ContainerDied","Data":"7723978d5daa9201a3078a7e34e842e0b4c794ef1610927634de33ff2c465e3e"}
Nov 03 20:26:25.187319 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.187176    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv" event={"ID":"7e760329-c9c8-4d93-8efb-8aaa938899b5","Type":"ContainerDied","Data":"312aeb20a2d5993190317597d1bdb36d5c1b86763616247af6b75055d14af58b"}
Nov 03 20:26:25.187319 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.187198    5591 scope.go:122] "RemoveContainer" containerID="7723978d5daa9201a3078a7e34e842e0b4c794ef1610927634de33ff2c465e3e"
Nov 03 20:26:25.201143 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.201119    5591 scope.go:122] "RemoveContainer" containerID="7723978d5daa9201a3078a7e34e842e0b4c794ef1610927634de33ff2c465e3e"
Nov 03 20:26:25.201454 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:26:25.201432    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7723978d5daa9201a3078a7e34e842e0b4c794ef1610927634de33ff2c465e3e\": container with ID starting with 7723978d5daa9201a3078a7e34e842e0b4c794ef1610927634de33ff2c465e3e not found: ID does not exist" containerID="7723978d5daa9201a3078a7e34e842e0b4c794ef1610927634de33ff2c465e3e"
Nov 03 20:26:25.201526 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.201463    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7723978d5daa9201a3078a7e34e842e0b4c794ef1610927634de33ff2c465e3e"} err="failed to get container status \"7723978d5daa9201a3078a7e34e842e0b4c794ef1610927634de33ff2c465e3e\": rpc error: code = NotFound desc = could not find container \"7723978d5daa9201a3078a7e34e842e0b4c794ef1610927634de33ff2c465e3e\": container with ID starting with 7723978d5daa9201a3078a7e34e842e0b4c794ef1610927634de33ff2c465e3e not found: ID does not exist"
Nov 03 20:26:25.215831 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.215808    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"]
Nov 03 20:26:25.221980 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.221956    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-660d95-openshift-default-66c7467465-xgffv"]
Nov 03 20:26:25.343894 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.343859    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:25.352294 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.352265    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e760329-c9c8-4d93-8efb-8aaa938899b5" path="/var/lib/kubelet/pods/7e760329-c9c8-4d93-8efb-8aaa938899b5/volumes"
Nov 03 20:26:25.635837 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.635810    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"]
Nov 03 20:26:25.636183 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:26:25.636160    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c55ca26_6f34_4632_bf90_ab132d63fe78.slice/crio-38f4e5ffa537c846bf2c1dc626d499681c397511d24b461db6e0d83498cdb564 WatchSource:0}: Error finding container 38f4e5ffa537c846bf2c1dc626d499681c397511d24b461db6e0d83498cdb564: Status 404 returned error can't find the container with id 38f4e5ffa537c846bf2c1dc626d499681c397511d24b461db6e0d83498cdb564
Nov 03 20:26:25.639680 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.639636    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:25.639781 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.639723    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:25.639781 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:25.639752    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:26.201662 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:26.201623    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn" event={"ID":"6c55ca26-6f34-4632-bf90-ab132d63fe78","Type":"ContainerStarted","Data":"0aaf48050cda6108afe015a5bd2a3cc7e1cf86df3b80369ca733592b19bf2bf4"}
Nov 03 20:26:26.202244 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:26.201682    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn" event={"ID":"6c55ca26-6f34-4632-bf90-ab132d63fe78","Type":"ContainerStarted","Data":"38f4e5ffa537c846bf2c1dc626d499681c397511d24b461db6e0d83498cdb564"}
Nov 03 20:26:26.232593 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:26.232545    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn" podStartSLOduration=66.23253039 podStartE2EDuration="1m6.23253039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:26.226135089 +0000 UTC m=+2934.998428117" watchObservedRunningTime="2026-11-03 20:26:26.23253039 +0000 UTC m=+2935.004823469"
Nov 03 20:26:26.344927 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:26.344893    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:26.346648 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:26.346619    5591 patch_prober.go:29] interesting pod/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body=
Nov 03 20:26:26.346782 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:26.346681    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn" podUID="6c55ca26-6f34-4632-bf90-ab132d63fe78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused"
Nov 03 20:26:27.344606 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:27.344568    5591 patch_prober.go:29] interesting pod/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body=
Nov 03 20:26:27.345140 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:27.344628    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn" podUID="6c55ca26-6f34-4632-bf90-ab132d63fe78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused"
Nov 03 20:26:28.344356 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.344322    5591 patch_prober.go:29] interesting pod/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body=
Nov 03 20:26:28.344528 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.344384    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn" podUID="6c55ca26-6f34-4632-bf90-ab132d63fe78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused"
Nov 03 20:26:28.397479 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:26:28.397435    5591 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb043886c_68e1_48d8_9ec4_8ecd4601a753.slice/crio-7caaeec4c314cef19e22813fb950cf5661b3f97fe0c26e48513fc643739a1b62.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:26:28.644346 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.644316    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 20:26:28.688792 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.688753    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"]
Nov 03 20:26:28.690263 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.690237    5591 cpu_manager.go:414] "containerMap: removing container" podUID="b043886c-68e1-48d8-9ec4-8ecd4601a753" containerName="istio-proxy"
Nov 03 20:26:28.690263 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.690262    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b043886c-68e1-48d8-9ec4-8ecd4601a753" containerName="istio-proxy"
Nov 03 20:26:28.690571 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.690553    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b043886c-68e1-48d8-9ec4-8ecd4601a753" containerName="istio-proxy"
Nov 03 20:26:28.700950 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.700921    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:28.708841 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.708812    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"]
Nov 03 20:26:28.710298 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.710271    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e7b70d-openshift-default-dockercfg-jjqkc\""
Nov 03 20:26:28.721797 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.721765    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-workload-certs\") pod \"b043886c-68e1-48d8-9ec4-8ecd4601a753\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") "
Nov 03 20:26:28.721947 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.721894    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-token\" (UniqueName: \"kubernetes.io/projected/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-token\") pod \"b043886c-68e1-48d8-9ec4-8ecd4601a753\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") "
Nov 03 20:26:28.722022 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.721990    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-data\") pod \"b043886c-68e1-48d8-9ec4-8ecd4601a753\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") "
Nov 03 20:26:28.722090 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.722078    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-podinfo\") pod \"b043886c-68e1-48d8-9ec4-8ecd4601a753\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") "
Nov 03 20:26:28.722149 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.722100    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-workload-certs" pod "b043886c-68e1-48d8-9ec4-8ecd4601a753" (UID: "b043886c-68e1-48d8-9ec4-8ecd4601a753"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:28.722149 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.722111    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-credential-socket\") pod \"b043886c-68e1-48d8-9ec4-8ecd4601a753\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") "
Nov 03 20:26:28.722333 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.722310    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b043886c-68e1-48d8-9ec4-8ecd4601a753-kube-api-access-lxqz2\" (UniqueName: \"kubernetes.io/projected/b043886c-68e1-48d8-9ec4-8ecd4601a753-kube-api-access-lxqz2\") pod \"b043886c-68e1-48d8-9ec4-8ecd4601a753\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") "
Nov 03 20:26:28.722415 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.722395    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-credential-socket" pod "b043886c-68e1-48d8-9ec4-8ecd4601a753" (UID: "b043886c-68e1-48d8-9ec4-8ecd4601a753"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:28.722537 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.722516    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-workload-socket\") pod \"b043886c-68e1-48d8-9ec4-8ecd4601a753\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") "
Nov 03 20:26:28.722608 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.722561    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b043886c-68e1-48d8-9ec4-8ecd4601a753-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b043886c-68e1-48d8-9ec4-8ecd4601a753-istiod-ca-cert\") pod \"b043886c-68e1-48d8-9ec4-8ecd4601a753\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") "
Nov 03 20:26:28.722608 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.722599    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-envoy\") pod \"b043886c-68e1-48d8-9ec4-8ecd4601a753\" (UID: \"b043886c-68e1-48d8-9ec4-8ecd4601a753\") "
Nov 03 20:26:28.722906 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.722884    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-data" pod "b043886c-68e1-48d8-9ec4-8ecd4601a753" (UID: "b043886c-68e1-48d8-9ec4-8ecd4601a753"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:28.723319 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.723293    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-workload-socket" pod "b043886c-68e1-48d8-9ec4-8ecd4601a753" (UID: "b043886c-68e1-48d8-9ec4-8ecd4601a753"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:28.723802 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.723745    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:28.723802 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.723772    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:28.723802 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.723789    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:28.723802 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.723803    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:28.724414 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.724150    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b043886c-68e1-48d8-9ec4-8ecd4601a753-istiod-ca-cert" pod "b043886c-68e1-48d8-9ec4-8ecd4601a753" (UID: "b043886c-68e1-48d8-9ec4-8ecd4601a753"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:28.727136 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.726425    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-token" pod "b043886c-68e1-48d8-9ec4-8ecd4601a753" (UID: "b043886c-68e1-48d8-9ec4-8ecd4601a753"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:28.727136 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.726638    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-podinfo" pod "b043886c-68e1-48d8-9ec4-8ecd4601a753" (UID: "b043886c-68e1-48d8-9ec4-8ecd4601a753"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:28.727136 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.726759    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b043886c-68e1-48d8-9ec4-8ecd4601a753-kube-api-access-lxqz2" pod "b043886c-68e1-48d8-9ec4-8ecd4601a753" (UID: "b043886c-68e1-48d8-9ec4-8ecd4601a753"). InnerVolumeSpecName "kube-api-access-lxqz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:28.727136 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.726920    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-envoy" pod "b043886c-68e1-48d8-9ec4-8ecd4601a753" (UID: "b043886c-68e1-48d8-9ec4-8ecd4601a753"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:28.745977 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:26:28.745944    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:26:28.825071 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.825039    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:28.825296 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.825077    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:28.825296 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.825128    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:28.825426 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.825307    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:28.825426 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.825358    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:28.825426 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.825420    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea34ecc5-6c75-44b0-a28a-530668e79e47-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:28.825567 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.825478    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:28.825725 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.825703    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:28.825922 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.825902    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22bvw\" (UniqueName: \"kubernetes.io/projected/ea34ecc5-6c75-44b0-a28a-530668e79e47-kube-api-access-22bvw\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:28.826094 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.826079    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxqz2\" (UniqueName: \"kubernetes.io/projected/b043886c-68e1-48d8-9ec4-8ecd4601a753-kube-api-access-lxqz2\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:28.826146 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.826100    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b043886c-68e1-48d8-9ec4-8ecd4601a753-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:28.826146 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.826115    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:28.826146 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.826135    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:28.826281 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.826149    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b043886c-68e1-48d8-9ec4-8ecd4601a753-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:28.927703 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.927605    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:28.927883 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.927757    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22bvw\" (UniqueName: \"kubernetes.io/projected/ea34ecc5-6c75-44b0-a28a-530668e79e47-kube-api-access-22bvw\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:28.927883 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.927824    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:28.927883 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.927856    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:28.928067 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.927924    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:28.928067 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.928004    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:28.928067 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.928043    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:28.928067 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.928063    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:28.928386 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.928128    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea34ecc5-6c75-44b0-a28a-530668e79e47-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:28.928386 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.928218    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:28.928386 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.928316    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:28.928386 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.928379    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:28.928650 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.928455    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:28.929149 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.929124    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea34ecc5-6c75-44b0-a28a-530668e79e47-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:28.930550 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.930520    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:28.930740 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.930723    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:28.941328 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.941304    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:28.941433 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:28.941410    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22bvw\" (UniqueName: \"kubernetes.io/projected/ea34ecc5-6c75-44b0-a28a-530668e79e47-kube-api-access-22bvw\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:29.026936 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:29.026899    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:29.232805 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:29.232702    5591 generic.go:356] "Generic (PLEG): container finished" podID="b043886c-68e1-48d8-9ec4-8ecd4601a753" containerID="7caaeec4c314cef19e22813fb950cf5661b3f97fe0c26e48513fc643739a1b62" exitCode=0
Nov 03 20:26:29.232805 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:29.232783    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"
Nov 03 20:26:29.232805 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:29.232792    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr" event={"ID":"b043886c-68e1-48d8-9ec4-8ecd4601a753","Type":"ContainerDied","Data":"7caaeec4c314cef19e22813fb950cf5661b3f97fe0c26e48513fc643739a1b62"}
Nov 03 20:26:29.233102 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:29.232884    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr" event={"ID":"b043886c-68e1-48d8-9ec4-8ecd4601a753","Type":"ContainerDied","Data":"2c8b6abb912158a274652b24b8b8ed0477ac587c6bd1aba78afcfd193ac2108f"}
Nov 03 20:26:29.233102 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:29.232906    5591 scope.go:122] "RemoveContainer" containerID="7caaeec4c314cef19e22813fb950cf5661b3f97fe0c26e48513fc643739a1b62"
Nov 03 20:26:29.245909 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:29.245888    5591 scope.go:122] "RemoveContainer" containerID="7caaeec4c314cef19e22813fb950cf5661b3f97fe0c26e48513fc643739a1b62"
Nov 03 20:26:29.246254 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:26:29.246231    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7caaeec4c314cef19e22813fb950cf5661b3f97fe0c26e48513fc643739a1b62\": container with ID starting with 7caaeec4c314cef19e22813fb950cf5661b3f97fe0c26e48513fc643739a1b62 not found: ID does not exist" containerID="7caaeec4c314cef19e22813fb950cf5661b3f97fe0c26e48513fc643739a1b62"
Nov 03 20:26:29.246339 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:29.246267    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7caaeec4c314cef19e22813fb950cf5661b3f97fe0c26e48513fc643739a1b62"} err="failed to get container status \"7caaeec4c314cef19e22813fb950cf5661b3f97fe0c26e48513fc643739a1b62\": rpc error: code = NotFound desc = could not find container \"7caaeec4c314cef19e22813fb950cf5661b3f97fe0c26e48513fc643739a1b62\": container with ID starting with 7caaeec4c314cef19e22813fb950cf5661b3f97fe0c26e48513fc643739a1b62 not found: ID does not exist"
Nov 03 20:26:29.265089 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:29.265064    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"]
Nov 03 20:26:29.269448 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:29.269426    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-9rntr"]
Nov 03 20:26:29.322752 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:26:29.322720    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea34ecc5_6c75_44b0_a28a_530668e79e47.slice/crio-d224895b6f5dba7bb321ee79639269aedc45aa9178b483fb282a0304409fb174 WatchSource:0}: Error finding container d224895b6f5dba7bb321ee79639269aedc45aa9178b483fb282a0304409fb174: Status 404 returned error can't find the container with id d224895b6f5dba7bb321ee79639269aedc45aa9178b483fb282a0304409fb174
Nov 03 20:26:29.323079 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:29.323058    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"]
Nov 03 20:26:29.326171 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:29.326132    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:29.326321 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:29.326222    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:29.326321 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:29.326269    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:29.344421 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:29.344385    5591 patch_prober.go:29] interesting pod/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body=
Nov 03 20:26:29.344572 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:29.344460    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn" podUID="6c55ca26-6f34-4632-bf90-ab132d63fe78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused"
Nov 03 20:26:29.355339 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:29.355312    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b043886c-68e1-48d8-9ec4-8ecd4601a753" path="/var/lib/kubelet/pods/b043886c-68e1-48d8-9ec4-8ecd4601a753/volumes"
Nov 03 20:26:30.244671 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:30.244636    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn" event={"ID":"ea34ecc5-6c75-44b0-a28a-530668e79e47","Type":"ContainerStarted","Data":"9bc089cd5e57f1afac4279e59f62b6e8f7bc7cc4671af1bc3079c826e3dcb246"}
Nov 03 20:26:30.245157 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:30.244712    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn" event={"ID":"ea34ecc5-6c75-44b0-a28a-530668e79e47","Type":"ContainerStarted","Data":"d224895b6f5dba7bb321ee79639269aedc45aa9178b483fb282a0304409fb174"}
Nov 03 20:26:30.294731 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:30.294664    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn" podStartSLOduration=101.294640249 podStartE2EDuration="1m41.294640249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:30.268909157 +0000 UTC m=+2939.041202176" watchObservedRunningTime="2026-11-03 20:26:30.294640249 +0000 UTC m=+2939.066933270"
Nov 03 20:26:30.344987 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:30.344957    5591 patch_prober.go:29] interesting pod/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body=
Nov 03 20:26:30.345262 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:30.345235    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn" podUID="6c55ca26-6f34-4632-bf90-ab132d63fe78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused"
Nov 03 20:26:31.027375 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:31.027330    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:31.029702 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:31.029667    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Nov 03 20:26:31.029870 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:31.029737    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn" podUID="ea34ecc5-6c75-44b0-a28a-530668e79e47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Nov 03 20:26:31.349473 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:31.349385    5591 patch_prober.go:29] interesting pod/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body=
Nov 03 20:26:31.349473 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:31.349448    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn" podUID="6c55ca26-6f34-4632-bf90-ab132d63fe78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused"
Nov 03 20:26:31.852050 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:31.852012    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"]
Nov 03 20:26:31.852282 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:31.852219    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs" podUID="76f4135b-c979-49ee-a372-a68759b98276" containerName="istio-proxy" containerID="cri-o://24ea9acfd556d3c8ae1ef8317cadacc3fe210c949e594f097deb58509723f02d" gracePeriod=30
Nov 03 20:26:32.027647 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:32.027610    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Nov 03 20:26:32.027797 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:32.027667    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn" podUID="ea34ecc5-6c75-44b0-a28a-530668e79e47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Nov 03 20:26:32.344345 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:32.344309    5591 patch_prober.go:29] interesting pod/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body=
Nov 03 20:26:32.344540 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:32.344371    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn" podUID="6c55ca26-6f34-4632-bf90-ab132d63fe78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused"
Nov 03 20:26:33.027762 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:33.027721    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Nov 03 20:26:33.028393 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:33.028357    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn" podUID="ea34ecc5-6c75-44b0-a28a-530668e79e47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Nov 03 20:26:33.344456 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:33.344370    5591 patch_prober.go:29] interesting pod/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body=
Nov 03 20:26:33.344456 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:33.344426    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn" podUID="6c55ca26-6f34-4632-bf90-ab132d63fe78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused"
Nov 03 20:26:34.029349 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:34.029310    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Nov 03 20:26:34.029965 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:34.029926    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn" podUID="ea34ecc5-6c75-44b0-a28a-530668e79e47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Nov 03 20:26:34.350591 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:34.350509    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:34.351011 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:34.350984    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:34.351814 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:34.351794    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:26:35.027982 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:35.027941    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body=
Nov 03 20:26:35.028328 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:35.028285    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn" podUID="ea34ecc5-6c75-44b0-a28a-530668e79e47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused"
Nov 03 20:26:35.355516 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:35.355443    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/odh-model-controller-769875db56-zpphn"]
Nov 03 20:26:35.356118 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:35.356077    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/odh-model-controller-769875db56-zpphn" podUID="d7b3e7a0-4984-4088-a297-be74c8796194" containerName="manager" containerID="cri-o://55649a7649ceb35243232d9fe172c1107b039989eab17949cacc7b9667d74125" gracePeriod=10
Nov 03 20:26:35.738410 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:35.738381    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-zpphn"
Nov 03 20:26:35.794802 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:35.794769    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7b3e7a0-4984-4088-a297-be74c8796194-kube-api-access-tvnvw\" (UniqueName: \"kubernetes.io/projected/d7b3e7a0-4984-4088-a297-be74c8796194-kube-api-access-tvnvw\") pod \"d7b3e7a0-4984-4088-a297-be74c8796194\" (UID: \"d7b3e7a0-4984-4088-a297-be74c8796194\") "
Nov 03 20:26:35.795022 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:35.794912    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7b3e7a0-4984-4088-a297-be74c8796194-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7b3e7a0-4984-4088-a297-be74c8796194-metrics-certs\") pod \"d7b3e7a0-4984-4088-a297-be74c8796194\" (UID: \"d7b3e7a0-4984-4088-a297-be74c8796194\") "
Nov 03 20:26:35.795022 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:35.794959    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7b3e7a0-4984-4088-a297-be74c8796194-cert\" (UniqueName: \"kubernetes.io/secret/d7b3e7a0-4984-4088-a297-be74c8796194-cert\") pod \"d7b3e7a0-4984-4088-a297-be74c8796194\" (UID: \"d7b3e7a0-4984-4088-a297-be74c8796194\") "
Nov 03 20:26:35.798953 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:35.798914    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7b3e7a0-4984-4088-a297-be74c8796194-metrics-certs" pod "d7b3e7a0-4984-4088-a297-be74c8796194" (UID: "d7b3e7a0-4984-4088-a297-be74c8796194"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:35.800697 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:35.800666    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7b3e7a0-4984-4088-a297-be74c8796194-cert" pod "d7b3e7a0-4984-4088-a297-be74c8796194" (UID: "d7b3e7a0-4984-4088-a297-be74c8796194"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:26:35.805778 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:35.805709    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b3e7a0-4984-4088-a297-be74c8796194-kube-api-access-tvnvw" pod "d7b3e7a0-4984-4088-a297-be74c8796194" (UID: "d7b3e7a0-4984-4088-a297-be74c8796194"). InnerVolumeSpecName "kube-api-access-tvnvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:35.897425 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:35.897392    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvnvw\" (UniqueName: \"kubernetes.io/projected/d7b3e7a0-4984-4088-a297-be74c8796194-kube-api-access-tvnvw\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.897425 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:35.897427    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7b3e7a0-4984-4088-a297-be74c8796194-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:35.897716 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:35.897444    5591 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7b3e7a0-4984-4088-a297-be74c8796194-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:36.084376 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:36.084342    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:36.086145 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:36.085668    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:36.087766 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:36.087740    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:26:36.332412 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:36.332378    5591 generic.go:356] "Generic (PLEG): container finished" podID="d7b3e7a0-4984-4088-a297-be74c8796194" containerID="55649a7649ceb35243232d9fe172c1107b039989eab17949cacc7b9667d74125" exitCode=0
Nov 03 20:26:36.332639 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:36.332430    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-zpphn"
Nov 03 20:26:36.332639 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:36.332465    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-zpphn" event={"ID":"d7b3e7a0-4984-4088-a297-be74c8796194","Type":"ContainerDied","Data":"55649a7649ceb35243232d9fe172c1107b039989eab17949cacc7b9667d74125"}
Nov 03 20:26:36.332639 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:36.332519    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-zpphn" event={"ID":"d7b3e7a0-4984-4088-a297-be74c8796194","Type":"ContainerDied","Data":"9661c2c8a9492505a1035407c959ae5ce9a65fe46d7023f0868cf0e2d437aada"}
Nov 03 20:26:36.332639 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:36.332542    5591 scope.go:122] "RemoveContainer" containerID="55649a7649ceb35243232d9fe172c1107b039989eab17949cacc7b9667d74125"
Nov 03 20:26:36.358498 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:36.358386    5591 scope.go:122] "RemoveContainer" containerID="55649a7649ceb35243232d9fe172c1107b039989eab17949cacc7b9667d74125"
Nov 03 20:26:36.358984 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:26:36.358958    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55649a7649ceb35243232d9fe172c1107b039989eab17949cacc7b9667d74125\": container with ID starting with 55649a7649ceb35243232d9fe172c1107b039989eab17949cacc7b9667d74125 not found: ID does not exist" containerID="55649a7649ceb35243232d9fe172c1107b039989eab17949cacc7b9667d74125"
Nov 03 20:26:36.359032 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:36.358994    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55649a7649ceb35243232d9fe172c1107b039989eab17949cacc7b9667d74125"} err="failed to get container status \"55649a7649ceb35243232d9fe172c1107b039989eab17949cacc7b9667d74125\": rpc error: code = NotFound desc = could not find container \"55649a7649ceb35243232d9fe172c1107b039989eab17949cacc7b9667d74125\": container with ID starting with 55649a7649ceb35243232d9fe172c1107b039989eab17949cacc7b9667d74125 not found: ID does not exist"
Nov 03 20:26:36.361016 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:36.360992    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/odh-model-controller-769875db56-zpphn"]
Nov 03 20:26:36.367248 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:36.367216    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/odh-model-controller-769875db56-zpphn"]
Nov 03 20:26:37.220195 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.220164    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 20:26:37.260116 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.260079    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"]
Nov 03 20:26:37.261343 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.261315    5591 cpu_manager.go:414] "containerMap: removing container" podUID="76f4135b-c979-49ee-a372-a68759b98276" containerName="istio-proxy"
Nov 03 20:26:37.261343 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.261342    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76f4135b-c979-49ee-a372-a68759b98276" containerName="istio-proxy"
Nov 03 20:26:37.261520 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.261383    5591 cpu_manager.go:414] "containerMap: removing container" podUID="d7b3e7a0-4984-4088-a297-be74c8796194" containerName="manager"
Nov 03 20:26:37.261520 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.261392    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7b3e7a0-4984-4088-a297-be74c8796194" containerName="manager"
Nov 03 20:26:37.261729 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.261710    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="76f4135b-c979-49ee-a372-a68759b98276" containerName="istio-proxy"
Nov 03 20:26:37.261766 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.261746    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7b3e7a0-4984-4088-a297-be74c8796194" containerName="manager"
Nov 03 20:26:37.265872 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.265852    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.286151 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.283783    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"]
Nov 03 20:26:37.332652 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.332620    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76f4135b-c979-49ee-a372-a68759b98276-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f4135b-c979-49ee-a372-a68759b98276-istiod-ca-cert\") pod \"76f4135b-c979-49ee-a372-a68759b98276\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") "
Nov 03 20:26:37.332851 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.332695    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-workload-certs\") pod \"76f4135b-c979-49ee-a372-a68759b98276\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") "
Nov 03 20:26:37.332851 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.332822    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-workload-socket\") pod \"76f4135b-c979-49ee-a372-a68759b98276\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") "
Nov 03 20:26:37.332851 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.332850    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76f4135b-c979-49ee-a372-a68759b98276-kube-api-access-dcvbg\" (UniqueName: \"kubernetes.io/projected/76f4135b-c979-49ee-a372-a68759b98276-kube-api-access-dcvbg\") pod \"76f4135b-c979-49ee-a372-a68759b98276\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") "
Nov 03 20:26:37.333046 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.332914    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-istio-envoy\") pod \"76f4135b-c979-49ee-a372-a68759b98276\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") "
Nov 03 20:26:37.333046 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.332998    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76f4135b-c979-49ee-a372-a68759b98276-istio-token\" (UniqueName: \"kubernetes.io/projected/76f4135b-c979-49ee-a372-a68759b98276-istio-token\") pod \"76f4135b-c979-49ee-a372-a68759b98276\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") "
Nov 03 20:26:37.333284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.333116    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-istio-data\") pod \"76f4135b-c979-49ee-a372-a68759b98276\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") "
Nov 03 20:26:37.333284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.333158    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-workload-socket" pod "76f4135b-c979-49ee-a372-a68759b98276" (UID: "76f4135b-c979-49ee-a372-a68759b98276"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:37.333284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.333170    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76f4135b-c979-49ee-a372-a68759b98276-istiod-ca-cert" pod "76f4135b-c979-49ee-a372-a68759b98276" (UID: "76f4135b-c979-49ee-a372-a68759b98276"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:26:37.333429 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.333319    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-credential-socket\") pod \"76f4135b-c979-49ee-a372-a68759b98276\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") "
Nov 03 20:26:37.333429 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.333405    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76f4135b-c979-49ee-a372-a68759b98276-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f4135b-c979-49ee-a372-a68759b98276-istio-podinfo\") pod \"76f4135b-c979-49ee-a372-a68759b98276\" (UID: \"76f4135b-c979-49ee-a372-a68759b98276\") "
Nov 03 20:26:37.335227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.333668    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-istio-data" pod "76f4135b-c979-49ee-a372-a68759b98276" (UID: "76f4135b-c979-49ee-a372-a68759b98276"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:37.335227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.333767    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.335227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.333922    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-credential-socket" pod "76f4135b-c979-49ee-a372-a68759b98276" (UID: "76f4135b-c979-49ee-a372-a68759b98276"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:37.335227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.333945    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f8ks\" (UniqueName: \"kubernetes.io/projected/3163904f-cc70-40d4-b196-563e2b900b29-kube-api-access-4f8ks\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.335227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.334067    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3163904f-cc70-40d4-b196-563e2b900b29-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.335227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.334092    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.335227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.334283    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.335227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.334426    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.335227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.334696    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3163904f-cc70-40d4-b196-563e2b900b29-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.335227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.334782    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.335227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.334834    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3163904f-cc70-40d4-b196-563e2b900b29-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.335227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.334982    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:37.335227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.335003    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f4135b-c979-49ee-a372-a68759b98276-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:37.335227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.335019    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:37.335227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.335033    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:37.336506 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.336484    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-workload-certs" pod "76f4135b-c979-49ee-a372-a68759b98276" (UID: "76f4135b-c979-49ee-a372-a68759b98276"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:37.338739 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.338709    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76f4135b-c979-49ee-a372-a68759b98276-istio-podinfo" pod "76f4135b-c979-49ee-a372-a68759b98276" (UID: "76f4135b-c979-49ee-a372-a68759b98276"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:26:37.338739 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.338834    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-istio-envoy" pod "76f4135b-c979-49ee-a372-a68759b98276" (UID: "76f4135b-c979-49ee-a372-a68759b98276"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:26:37.338739 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.339843    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f4135b-c979-49ee-a372-a68759b98276-istio-token" pod "76f4135b-c979-49ee-a372-a68759b98276" (UID: "76f4135b-c979-49ee-a372-a68759b98276"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:37.342283 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.341965    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f4135b-c979-49ee-a372-a68759b98276-kube-api-access-dcvbg" pod "76f4135b-c979-49ee-a372-a68759b98276" (UID: "76f4135b-c979-49ee-a372-a68759b98276"). InnerVolumeSpecName "kube-api-access-dcvbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:26:37.346910 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.346887    5591 generic.go:356] "Generic (PLEG): container finished" podID="76f4135b-c979-49ee-a372-a68759b98276" containerID="24ea9acfd556d3c8ae1ef8317cadacc3fe210c949e594f097deb58509723f02d" exitCode=0
Nov 03 20:26:37.347028 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.346993    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 20:26:37.355672 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.355649    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7b3e7a0-4984-4088-a297-be74c8796194" path="/var/lib/kubelet/pods/d7b3e7a0-4984-4088-a297-be74c8796194/volumes"
Nov 03 20:26:37.356305 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.356286    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs" event={"ID":"76f4135b-c979-49ee-a372-a68759b98276","Type":"ContainerDied","Data":"24ea9acfd556d3c8ae1ef8317cadacc3fe210c949e594f097deb58509723f02d"}
Nov 03 20:26:37.356396 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.356381    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs" event={"ID":"76f4135b-c979-49ee-a372-a68759b98276","Type":"ContainerDied","Data":"d2ec0abadaf7c8323afb904cf32ce3692bf439230cc24222aee7c4f3697ca536"}
Nov 03 20:26:37.356438 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.356413    5591 scope.go:122] "RemoveContainer" containerID="24ea9acfd556d3c8ae1ef8317cadacc3fe210c949e594f097deb58509723f02d"
Nov 03 20:26:37.381776 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.381748    5591 scope.go:122] "RemoveContainer" containerID="24ea9acfd556d3c8ae1ef8317cadacc3fe210c949e594f097deb58509723f02d"
Nov 03 20:26:37.382158 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:26:37.382109    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24ea9acfd556d3c8ae1ef8317cadacc3fe210c949e594f097deb58509723f02d\": container with ID starting with 24ea9acfd556d3c8ae1ef8317cadacc3fe210c949e594f097deb58509723f02d not found: ID does not exist" containerID="24ea9acfd556d3c8ae1ef8317cadacc3fe210c949e594f097deb58509723f02d"
Nov 03 20:26:37.382158 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.382133    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24ea9acfd556d3c8ae1ef8317cadacc3fe210c949e594f097deb58509723f02d"} err="failed to get container status \"24ea9acfd556d3c8ae1ef8317cadacc3fe210c949e594f097deb58509723f02d\": rpc error: code = NotFound desc = could not find container \"24ea9acfd556d3c8ae1ef8317cadacc3fe210c949e594f097deb58509723f02d\": container with ID starting with 24ea9acfd556d3c8ae1ef8317cadacc3fe210c949e594f097deb58509723f02d not found: ID does not exist"
Nov 03 20:26:37.437263 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.437154    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3163904f-cc70-40d4-b196-563e2b900b29-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.437436 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.437289    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.437436 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.437341    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3163904f-cc70-40d4-b196-563e2b900b29-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.437436 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.437413    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.437576 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.437491    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f8ks\" (UniqueName: \"kubernetes.io/projected/3163904f-cc70-40d4-b196-563e2b900b29-kube-api-access-4f8ks\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.437643 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.437607    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3163904f-cc70-40d4-b196-563e2b900b29-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.437698 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.437644    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.437755 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.437733    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.437814 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.437768    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.437882 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.437851    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.437968 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.437947    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.438385 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.438075    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.438385 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.438227    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:37.438385 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.438253    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcvbg\" (UniqueName: \"kubernetes.io/projected/76f4135b-c979-49ee-a372-a68759b98276-kube-api-access-dcvbg\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:37.438385 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.438271    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f4135b-c979-49ee-a372-a68759b98276-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:37.438385 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.438286    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f4135b-c979-49ee-a372-a68759b98276-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:37.438385 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.438301    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f4135b-c979-49ee-a372-a68759b98276-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:26:37.438385 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.438363    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3163904f-cc70-40d4-b196-563e2b900b29-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.439096 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.438677    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.440556 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.440505    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.441873 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.441852    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3163904f-cc70-40d4-b196-563e2b900b29-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.452566 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.452523    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f8ks\" (UniqueName: \"kubernetes.io/projected/3163904f-cc70-40d4-b196-563e2b900b29-kube-api-access-4f8ks\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.454175 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.454129    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3163904f-cc70-40d4-b196-563e2b900b29-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.589037 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.589005    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:37.891558 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:26:37.891523    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3163904f_cc70_40d4_b196_563e2b900b29.slice/crio-a294e8820faaecc04182a50dd6fb22f9e8d21f849c4f8590ad1f36343be7f43d WatchSource:0}: Error finding container a294e8820faaecc04182a50dd6fb22f9e8d21f849c4f8590ad1f36343be7f43d: Status 404 returned error can't find the container with id a294e8820faaecc04182a50dd6fb22f9e8d21f849c4f8590ad1f36343be7f43d
Nov 03 20:26:37.892505 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.892481    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"]
Nov 03 20:26:37.895605 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.895561    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:37.895704 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.895660    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:37.895769 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:37.895706    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:26:38.360850 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:38.360756    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" event={"ID":"3163904f-cc70-40d4-b196-563e2b900b29","Type":"ContainerStarted","Data":"c682b4964142f5e0b9b5103c615b3529905b71dd682fccfc7e7593ab45717cf3"}
Nov 03 20:26:38.360850 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:38.360821    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" event={"ID":"3163904f-cc70-40d4-b196-563e2b900b29","Type":"ContainerStarted","Data":"a294e8820faaecc04182a50dd6fb22f9e8d21f849c4f8590ad1f36343be7f43d"}
Nov 03 20:26:38.589241 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:38.589190    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:39.591708 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:39.591655    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 03 20:26:39.592257 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:39.591727    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 03 20:26:40.590420 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:40.590380    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 03 20:26:40.590615 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:40.590457    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 03 20:26:41.589882 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:41.589849    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 03 20:26:41.590326 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:41.589912    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 03 20:26:41.822413 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:41.822361    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podStartSLOduration=172.822345732 podStartE2EDuration="2m52.822345732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:38.389836833 +0000 UTC m=+2947.162129876" watchObservedRunningTime="2026-11-03 20:26:41.822345732 +0000 UTC m=+2950.594638750"
Nov 03 20:26:41.822594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:41.822581    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-w9ppp"]
Nov 03 20:26:41.829432 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:41.829407    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-w9ppp"
Nov 03 20:26:41.837499 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:41.837470    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 03 20:26:41.837676 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:41.837596    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 03 20:26:41.837676 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:41.837659    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-74fgb\""
Nov 03 20:26:41.839569 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:41.839545    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-w9ppp"]
Nov 03 20:26:41.914369 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:41.914331    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d-cert\") pod \"odh-model-controller-769875db56-w9ppp\" (UID: \"55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d\") " pod="opendatahub/odh-model-controller-769875db56-w9ppp"
Nov 03 20:26:41.914527 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:41.914496    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vczd\" (UniqueName: \"kubernetes.io/projected/55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d-kube-api-access-5vczd\") pod \"odh-model-controller-769875db56-w9ppp\" (UID: \"55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d\") " pod="opendatahub/odh-model-controller-769875db56-w9ppp"
Nov 03 20:26:41.914593 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:41.914561    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d-metrics-certs\") pod \"odh-model-controller-769875db56-w9ppp\" (UID: \"55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d\") " pod="opendatahub/odh-model-controller-769875db56-w9ppp"
Nov 03 20:26:42.016139 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:42.016097    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vczd\" (UniqueName: \"kubernetes.io/projected/55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d-kube-api-access-5vczd\") pod \"odh-model-controller-769875db56-w9ppp\" (UID: \"55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d\") " pod="opendatahub/odh-model-controller-769875db56-w9ppp"
Nov 03 20:26:42.016352 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:42.016153    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d-metrics-certs\") pod \"odh-model-controller-769875db56-w9ppp\" (UID: \"55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d\") " pod="opendatahub/odh-model-controller-769875db56-w9ppp"
Nov 03 20:26:42.016471 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:42.016453    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d-cert\") pod \"odh-model-controller-769875db56-w9ppp\" (UID: \"55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d\") " pod="opendatahub/odh-model-controller-769875db56-w9ppp"
Nov 03 20:26:42.018967 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:42.018946    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d-cert\") pod \"odh-model-controller-769875db56-w9ppp\" (UID: \"55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d\") " pod="opendatahub/odh-model-controller-769875db56-w9ppp"
Nov 03 20:26:42.019086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:42.019002    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d-metrics-certs\") pod \"odh-model-controller-769875db56-w9ppp\" (UID: \"55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d\") " pod="opendatahub/odh-model-controller-769875db56-w9ppp"
Nov 03 20:26:42.028386 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:42.028361    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vczd\" (UniqueName: \"kubernetes.io/projected/55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d-kube-api-access-5vczd\") pod \"odh-model-controller-769875db56-w9ppp\" (UID: \"55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d\") " pod="opendatahub/odh-model-controller-769875db56-w9ppp"
Nov 03 20:26:42.151659 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:42.151576    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-w9ppp"
Nov 03 20:26:42.450481 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:26:42.450446    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55ccd33d_4c5c_4d9b_8a54_e1ca4b48b24d.slice/crio-6523f04611daf97cbb6737e5422a2736d5a7f4c915c97fc13d5d43e64abe58d6 WatchSource:0}: Error finding container 6523f04611daf97cbb6737e5422a2736d5a7f4c915c97fc13d5d43e64abe58d6: Status 404 returned error can't find the container with id 6523f04611daf97cbb6737e5422a2736d5a7f4c915c97fc13d5d43e64abe58d6
Nov 03 20:26:42.451063 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:42.450959    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-w9ppp"]
Nov 03 20:26:42.589817 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:42.589781    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 03 20:26:42.589998 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:42.589846    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 03 20:26:43.414476 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:43.414440    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-w9ppp" event={"ID":"55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d","Type":"ContainerStarted","Data":"abba62fc47336c1b1f46f30566594de1ec1b62b992313b2f933a031d60410ffc"}
Nov 03 20:26:43.414732 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:43.414504    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-w9ppp" event={"ID":"55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d","Type":"ContainerStarted","Data":"6523f04611daf97cbb6737e5422a2736d5a7f4c915c97fc13d5d43e64abe58d6"}
Nov 03 20:26:43.414837 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:43.414819    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-w9ppp"
Nov 03 20:26:43.442974 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:43.442920    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-w9ppp" podStartSLOduration=1.882406207 podStartE2EDuration="2.44290447s" totalImagesPullingTime="560.498263ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:43.441956701 +0000 UTC m=+2952.214249720" watchObservedRunningTime="2026-11-03 20:26:43.44290447 +0000 UTC m=+2952.215197488"
Nov 03 20:26:43.589724 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:43.589692    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 03 20:26:43.589917 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:43.589744    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 03 20:26:44.589601 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:44.589563    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 03 20:26:44.590020 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:44.589623    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 03 20:26:45.589678 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:45.589644    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 03 20:26:45.590100 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:45.589697    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 03 20:26:46.589553 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:46.589515    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 03 20:26:46.589892 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:46.589583    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 03 20:26:47.590148 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:47.590109    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 03 20:26:47.590718 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:47.590180    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 03 20:26:48.589953 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:48.589922    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 03 20:26:48.590149 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:48.589976    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 03 20:26:49.590211 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:49.590157    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 03 20:26:49.590631 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:49.590258    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 03 20:26:50.589569 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:50.589536    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 03 20:26:50.589763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:50.589604    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 03 20:26:51.589625 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:51.589594    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 03 20:26:51.590019 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:51.589657    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 03 20:26:52.590044 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:52.589966    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:26:52.590498 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:52.590192    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 03 20:26:52.590498 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:52.590286    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 03 20:26:53.589365 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:53.589332    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 03 20:26:53.589565 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:53.589384    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 03 20:26:54.424342 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:54.424312    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-w9ppp"
Nov 03 20:26:54.589371 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:54.589336    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 03 20:26:54.589676 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:54.589398    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 03 20:26:55.589749 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:55.589709    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 03 20:26:55.590156 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:55.589782    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 03 20:26:56.589729 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:56.589698    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 03 20:26:56.589931 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:56.589751    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 03 20:26:57.589914 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:57.589877    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 03 20:26:57.590102 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:57.589933    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 03 20:26:58.589378 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:58.589341    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 03 20:26:58.589573 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:58.589408    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 03 20:26:58.726262 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:26:58.726219    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:26:59.589487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:59.589444    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 03 20:26:59.589711 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:26:59.589504    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 03 20:27:00.604010 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:00.603973    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 03 20:27:00.604653 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:00.604616    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 03 20:27:01.601244 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:01.600328    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 03 20:27:01.601244 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:01.600403    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 03 20:27:02.613987 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:02.613942    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 03 20:27:02.617623 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:02.617568    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 03 20:27:03.606236 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:03.602501    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 03 20:27:03.606568 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:03.606528    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 03 20:27:04.589588 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:04.589544    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 03 20:27:04.590283 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:04.590250    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 03 20:27:05.599898 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:05.599858    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused" start-of-body=
Nov 03 20:27:05.600614 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:05.600579    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.253:15021/healthz/ready\": dial tcp 10.134.0.253:15021: connect: connection refused"
Nov 03 20:27:06.595167 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:06.595138    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:27:06.596191 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:06.596172    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:27:07.374882 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:07.374847    5591 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod76f4135b-c979-49ee-a372-a68759b98276"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod76f4135b-c979-49ee-a372-a68759b98276] : Timed out while waiting for systemd to remove kubepods-burstable-pod76f4135b_c979_49ee_a372_a68759b98276.slice"
Nov 03 20:27:07.374882 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:27:07.374882    5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods burstable pod76f4135b-c979-49ee-a372-a68759b98276] : unable to destroy cgroup paths for cgroup [kubepods burstable pod76f4135b-c979-49ee-a372-a68759b98276] : Timed out while waiting for systemd to remove kubepods-burstable-pod76f4135b_c979_49ee_a372_a68759b98276.slice" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs" podUID="76f4135b-c979-49ee-a372-a68759b98276"
Nov 03 20:27:07.724074 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:07.723999    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"
Nov 03 20:27:07.778665 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:07.778633    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"]
Nov 03 20:27:07.788236 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:07.788174    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-tq8rs"]
Nov 03 20:27:09.352671 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:09.352630    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76f4135b-c979-49ee-a372-a68759b98276" path="/var/lib/kubelet/pods/76f4135b-c979-49ee-a372-a68759b98276/volumes"
Nov 03 20:27:28.727138 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:27:28.727098    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:27:31.013314 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:31.013279    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"]
Nov 03 20:27:31.013911 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:31.013572    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn" podUID="6c55ca26-6f34-4632-bf90-ab132d63fe78" containerName="istio-proxy" containerID="cri-o://0aaf48050cda6108afe015a5bd2a3cc7e1cf86df3b80369ca733592b19bf2bf4" gracePeriod=30
Nov 03 20:27:33.924440 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:33.924412    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 20:27:33.924937 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:33.924916    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 20:27:34.353560 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:34.353510    5591 patch_prober.go:29] interesting pod/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:27:34.353740 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:34.353600    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn" podUID="6c55ca26-6f34-4632-bf90-ab132d63fe78" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:27:36.332479 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.332450    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:27:36.373649 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.373615    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"]
Nov 03 20:27:36.374648 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.374624    5591 cpu_manager.go:414] "containerMap: removing container" podUID="6c55ca26-6f34-4632-bf90-ab132d63fe78" containerName="istio-proxy"
Nov 03 20:27:36.374648 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.374647    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c55ca26-6f34-4632-bf90-ab132d63fe78" containerName="istio-proxy"
Nov 03 20:27:36.374934 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.374918    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c55ca26-6f34-4632-bf90-ab132d63fe78" containerName="istio-proxy"
Nov 03 20:27:36.379153 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.379127    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.406651 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.406617    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"]
Nov 03 20:27:36.474445 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.474410    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-workload-socket\") pod \"6c55ca26-6f34-4632-bf90-ab132d63fe78\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") "
Nov 03 20:27:36.474445 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.474443    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-credential-socket\") pod \"6c55ca26-6f34-4632-bf90-ab132d63fe78\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") "
Nov 03 20:27:36.474715 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.474556    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-podinfo\") pod \"6c55ca26-6f34-4632-bf90-ab132d63fe78\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") "
Nov 03 20:27:36.474715 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.474614    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-workload-certs\") pod \"6c55ca26-6f34-4632-bf90-ab132d63fe78\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") "
Nov 03 20:27:36.474833 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.474729    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-workload-socket" pod "6c55ca26-6f34-4632-bf90-ab132d63fe78" (UID: "6c55ca26-6f34-4632-bf90-ab132d63fe78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:36.474833 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.474747    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c55ca26-6f34-4632-bf90-ab132d63fe78-kube-api-access-6dlng\" (UniqueName: \"kubernetes.io/projected/6c55ca26-6f34-4632-bf90-ab132d63fe78-kube-api-access-6dlng\") pod \"6c55ca26-6f34-4632-bf90-ab132d63fe78\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") "
Nov 03 20:27:36.474833 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.474805    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-credential-socket" pod "6c55ca26-6f34-4632-bf90-ab132d63fe78" (UID: "6c55ca26-6f34-4632-bf90-ab132d63fe78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:36.474985 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.474868    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-token\" (UniqueName: \"kubernetes.io/projected/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-token\") pod \"6c55ca26-6f34-4632-bf90-ab132d63fe78\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") "
Nov 03 20:27:36.474985 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.474880    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-workload-certs" pod "6c55ca26-6f34-4632-bf90-ab132d63fe78" (UID: "6c55ca26-6f34-4632-bf90-ab132d63fe78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:36.474985 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.474889    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-data\") pod \"6c55ca26-6f34-4632-bf90-ab132d63fe78\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") "
Nov 03 20:27:36.474985 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.474910    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-envoy\") pod \"6c55ca26-6f34-4632-bf90-ab132d63fe78\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") "
Nov 03 20:27:36.475192 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.475007    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c55ca26-6f34-4632-bf90-ab132d63fe78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c55ca26-6f34-4632-bf90-ab132d63fe78-istiod-ca-cert\") pod \"6c55ca26-6f34-4632-bf90-ab132d63fe78\" (UID: \"6c55ca26-6f34-4632-bf90-ab132d63fe78\") "
Nov 03 20:27:36.475192 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.475082    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-data" pod "6c55ca26-6f34-4632-bf90-ab132d63fe78" (UID: "6c55ca26-6f34-4632-bf90-ab132d63fe78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:36.475529 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.475479    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c55ca26-6f34-4632-bf90-ab132d63fe78-istiod-ca-cert" pod "6c55ca26-6f34-4632-bf90-ab132d63fe78" (UID: "6c55ca26-6f34-4632-bf90-ab132d63fe78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:27:36.475825 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.475785    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.475919 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.475899    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.475995 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.475943    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vprt2\" (UniqueName: \"kubernetes.io/projected/f51a1214-6af0-4ce3-b924-ca8d06c860fc-kube-api-access-vprt2\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.476059 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.476035    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.476129 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.476111    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.476314 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.476290    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.476444 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.476333    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.476444 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.476420    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.476653 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.476635    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.476854 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.476834    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:36.477028 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.476859    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c55ca26-6f34-4632-bf90-ab132d63fe78-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:36.477028 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.476873    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:36.477028 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.476891    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:36.477028 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.476903    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:36.478969 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.478936    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-token" pod "6c55ca26-6f34-4632-bf90-ab132d63fe78" (UID: "6c55ca26-6f34-4632-bf90-ab132d63fe78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:36.479057 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.478968    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c55ca26-6f34-4632-bf90-ab132d63fe78-kube-api-access-6dlng" pod "6c55ca26-6f34-4632-bf90-ab132d63fe78" (UID: "6c55ca26-6f34-4632-bf90-ab132d63fe78"). InnerVolumeSpecName "kube-api-access-6dlng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:27:36.479057 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.478994    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-envoy" pod "6c55ca26-6f34-4632-bf90-ab132d63fe78" (UID: "6c55ca26-6f34-4632-bf90-ab132d63fe78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:27:36.479057 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.479001    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-podinfo" pod "6c55ca26-6f34-4632-bf90-ab132d63fe78" (UID: "6c55ca26-6f34-4632-bf90-ab132d63fe78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:27:36.577956 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.577866    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.578130 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.577997    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.578306 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.578144    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.578306 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.578174    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.578306 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.578216    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vprt2\" (UniqueName: \"kubernetes.io/projected/f51a1214-6af0-4ce3-b924-ca8d06c860fc-kube-api-access-vprt2\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.578306 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.578257    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.578578 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.578318    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.578578 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.578316    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-data\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.578578 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.578415    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-workload-certs\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.578578 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.578415    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.578578 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.578486    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.578578 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.578536    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-workload-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.578888 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.578606    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:36.578888 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.578622    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dlng\" (UniqueName: \"kubernetes.io/projected/6c55ca26-6f34-4632-bf90-ab132d63fe78-kube-api-access-6dlng\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:36.578888 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.578637    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:36.578888 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.578650    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c55ca26-6f34-4632-bf90-ab132d63fe78-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:27:36.578888 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.578713    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-credential-socket\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.579301 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.579252    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istiod-ca-cert\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.581441 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.581409    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-envoy\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.581598 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.581577    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-podinfo\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.592319 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.592294    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-token\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.592434 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.592393    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vprt2\" (UniqueName: \"kubernetes.io/projected/f51a1214-6af0-4ce3-b924-ca8d06c860fc-kube-api-access-vprt2\") pod \"e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") " pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:36.710797 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:36.710765    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:37.010752 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:37.010722    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"]
Nov 03 20:27:37.011149 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:27:37.011124    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51a1214_6af0_4ce3_b924_ca8d06c860fc.slice/crio-0d55072d21cabe3937c04ac8cea92d1b6d2ee2f84f41bac3bdc2d6985e22fa62 WatchSource:0}: Error finding container 0d55072d21cabe3937c04ac8cea92d1b6d2ee2f84f41bac3bdc2d6985e22fa62: Status 404 returned error can't find the container with id 0d55072d21cabe3937c04ac8cea92d1b6d2ee2f84f41bac3bdc2d6985e22fa62
Nov 03 20:27:37.013966 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:37.013932    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:27:37.014047 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:37.014010    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:27:37.014047 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:37.014042    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:27:37.059574 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:37.059544    5591 generic.go:356] "Generic (PLEG): container finished" podID="6c55ca26-6f34-4632-bf90-ab132d63fe78" containerID="0aaf48050cda6108afe015a5bd2a3cc7e1cf86df3b80369ca733592b19bf2bf4" exitCode=0
Nov 03 20:27:37.059677 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:37.059595    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"
Nov 03 20:27:37.059677 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:37.059624    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn" event={"ID":"6c55ca26-6f34-4632-bf90-ab132d63fe78","Type":"ContainerDied","Data":"0aaf48050cda6108afe015a5bd2a3cc7e1cf86df3b80369ca733592b19bf2bf4"}
Nov 03 20:27:37.059780 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:37.059714    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn" event={"ID":"6c55ca26-6f34-4632-bf90-ab132d63fe78","Type":"ContainerDied","Data":"38f4e5ffa537c846bf2c1dc626d499681c397511d24b461db6e0d83498cdb564"}
Nov 03 20:27:37.059780 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:37.059744    5591 scope.go:122] "RemoveContainer" containerID="0aaf48050cda6108afe015a5bd2a3cc7e1cf86df3b80369ca733592b19bf2bf4"
Nov 03 20:27:37.061366 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:37.061340    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s" event={"ID":"f51a1214-6af0-4ce3-b924-ca8d06c860fc","Type":"ContainerStarted","Data":"0d55072d21cabe3937c04ac8cea92d1b6d2ee2f84f41bac3bdc2d6985e22fa62"}
Nov 03 20:27:37.076078 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:37.076056    5591 scope.go:122] "RemoveContainer" containerID="0aaf48050cda6108afe015a5bd2a3cc7e1cf86df3b80369ca733592b19bf2bf4"
Nov 03 20:27:37.076352 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:27:37.076333    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aaf48050cda6108afe015a5bd2a3cc7e1cf86df3b80369ca733592b19bf2bf4\": container with ID starting with 0aaf48050cda6108afe015a5bd2a3cc7e1cf86df3b80369ca733592b19bf2bf4 not found: ID does not exist" containerID="0aaf48050cda6108afe015a5bd2a3cc7e1cf86df3b80369ca733592b19bf2bf4"
Nov 03 20:27:37.076440 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:37.076362    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aaf48050cda6108afe015a5bd2a3cc7e1cf86df3b80369ca733592b19bf2bf4"} err="failed to get container status \"0aaf48050cda6108afe015a5bd2a3cc7e1cf86df3b80369ca733592b19bf2bf4\": rpc error: code = NotFound desc = could not find container \"0aaf48050cda6108afe015a5bd2a3cc7e1cf86df3b80369ca733592b19bf2bf4\": container with ID starting with 0aaf48050cda6108afe015a5bd2a3cc7e1cf86df3b80369ca733592b19bf2bf4 not found: ID does not exist"
Nov 03 20:27:37.096909 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:37.096876    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"]
Nov 03 20:27:37.103077 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:37.103053    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-759b04-openshift-default-5549dc77fc-gn8qn"]
Nov 03 20:27:37.402690 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:37.402650    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c55ca26-6f34-4632-bf90-ab132d63fe78" path="/var/lib/kubelet/pods/6c55ca26-6f34-4632-bf90-ab132d63fe78/volumes"
Nov 03 20:27:38.073994 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:38.073952    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s" event={"ID":"f51a1214-6af0-4ce3-b924-ca8d06c860fc","Type":"ContainerStarted","Data":"93fa05cccc54e0d6a2772a1eb9120569c678c347d5d5600b78af40654c5b06b9"}
Nov 03 20:27:38.113835 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:38.113778    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s" podStartSLOduration=214.113762676 podStartE2EDuration="3m34.113762676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:38.111583308 +0000 UTC m=+3006.883876324" watchObservedRunningTime="2026-11-03 20:27:38.113762676 +0000 UTC m=+3006.886055694"
Nov 03 20:27:38.711986 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:38.711946    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:38.714314 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:38.714285    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Nov 03 20:27:38.714464 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:38.714338    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s" podUID="f51a1214-6af0-4ce3-b924-ca8d06c860fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Nov 03 20:27:39.711489 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:39.711457    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Nov 03 20:27:39.711687 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:39.711526    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s" podUID="f51a1214-6af0-4ce3-b924-ca8d06c860fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Nov 03 20:27:40.711705 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:40.711666    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Nov 03 20:27:40.712293 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:40.712252    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s" podUID="f51a1214-6af0-4ce3-b924-ca8d06c860fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Nov 03 20:27:41.711291 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:41.711252    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Nov 03 20:27:41.711466 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:41.711308    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s" podUID="f51a1214-6af0-4ce3-b924-ca8d06c860fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Nov 03 20:27:42.711545 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:42.711507    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Nov 03 20:27:42.712004 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:42.711574    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s" podUID="f51a1214-6af0-4ce3-b924-ca8d06c860fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Nov 03 20:27:43.714291 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:43.714252    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body=
Nov 03 20:27:43.714812 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:43.714323    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s" podUID="f51a1214-6af0-4ce3-b924-ca8d06c860fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused"
Nov 03 20:27:44.717289 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:44.717260    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:44.717771 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:44.717755    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:44.718511 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:27:44.718488    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:27:58.726941 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:27:58.726902    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:28:11.372231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:11.359907    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl" podUID="97472d8f-eb02-4974-9a3d-275a65ca555b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.247:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:11.487244 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:11.473613    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87" podUID="666c0175-cb77-4af6-8793-81a7a684c1f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.249:8000/ready\": context deadline exceeded"
Nov 03 20:28:11.510260 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:11.504688    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:11.549243 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:11.542732    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:11.549243 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:11.543949    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:11.564254 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:11.553293    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:11.806154 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:11.806110    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-mrwmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:28:11.806482 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:11.806446    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt" podUID="e14ca8f3-c80b-4c0b-9564-ef9734b86de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:11.807026 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:11.806935    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:11.824153 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:11.823310    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:11.837883 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:11.836935    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:11.891370 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:11.891312    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:28:22.618438 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:22.618394    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf"]
Nov 03 20:28:27.618752 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:27.618705    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf" podUID="1b9f9140-739a-4f76-9165-6af845847971" containerName="payload-processing" containerID="cri-o://a16e5529c29277947c6f685c5b2e0ea1300314b9dfabeeb54f6ec4fc19c8724c" gracePeriod=25
Nov 03 20:28:28.004264 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:28.001147    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf"
Nov 03 20:28:28.017179 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:28.017139    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b9f9140-739a-4f76-9165-6af845847971-kube-api-access-cxs82\" (UniqueName: \"kubernetes.io/projected/1b9f9140-739a-4f76-9165-6af845847971-kube-api-access-cxs82\") pod \"1b9f9140-739a-4f76-9165-6af845847971\" (UID: \"1b9f9140-739a-4f76-9165-6af845847971\") "
Nov 03 20:28:28.017388 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:28.017301    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b9f9140-739a-4f76-9165-6af845847971-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b9f9140-739a-4f76-9165-6af845847971-provider-credentials\") pod \"1b9f9140-739a-4f76-9165-6af845847971\" (UID: \"1b9f9140-739a-4f76-9165-6af845847971\") "
Nov 03 20:28:28.019681 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:28.019656    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9f9140-739a-4f76-9165-6af845847971-provider-credentials" pod "1b9f9140-739a-4f76-9165-6af845847971" (UID: "1b9f9140-739a-4f76-9165-6af845847971"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:28:28.019803 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:28.019697    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9f9140-739a-4f76-9165-6af845847971-kube-api-access-cxs82" pod "1b9f9140-739a-4f76-9165-6af845847971" (UID: "1b9f9140-739a-4f76-9165-6af845847971"). InnerVolumeSpecName "kube-api-access-cxs82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:28:28.108884 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:28.108851    5591 generic.go:356] "Generic (PLEG): container finished" podID="1b9f9140-739a-4f76-9165-6af845847971" containerID="a16e5529c29277947c6f685c5b2e0ea1300314b9dfabeeb54f6ec4fc19c8724c" exitCode=0
Nov 03 20:28:28.109097 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:28.108921    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf"
Nov 03 20:28:28.109097 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:28.108939    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf" event={"ID":"1b9f9140-739a-4f76-9165-6af845847971","Type":"ContainerDied","Data":"a16e5529c29277947c6f685c5b2e0ea1300314b9dfabeeb54f6ec4fc19c8724c"}
Nov 03 20:28:28.109097 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:28.109028    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf" event={"ID":"1b9f9140-739a-4f76-9165-6af845847971","Type":"ContainerDied","Data":"6a642878dc6307e427c0addb91899aedf86ce0134ffe9f3e8f4ab2b38b06d5d5"}
Nov 03 20:28:28.109097 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:28.109059    5591 scope.go:122] "RemoveContainer" containerID="a16e5529c29277947c6f685c5b2e0ea1300314b9dfabeeb54f6ec4fc19c8724c"
Nov 03 20:28:28.119316 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:28.119283    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b9f9140-739a-4f76-9165-6af845847971-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b9f9140-739a-4f76-9165-6af845847971-routing-overlay-volume\") pod \"1b9f9140-739a-4f76-9165-6af845847971\" (UID: \"1b9f9140-739a-4f76-9165-6af845847971\") "
Nov 03 20:28:28.119723 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:28.119701    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b9f9140-739a-4f76-9165-6af845847971-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9f9140-739a-4f76-9165-6af845847971-plugins-config-volume\") pod \"1b9f9140-739a-4f76-9165-6af845847971\" (UID: \"1b9f9140-739a-4f76-9165-6af845847971\") "
Nov 03 20:28:28.120438 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:28.120410    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxs82\" (UniqueName: \"kubernetes.io/projected/1b9f9140-739a-4f76-9165-6af845847971-kube-api-access-cxs82\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:28.120552 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:28.120443    5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b9f9140-739a-4f76-9165-6af845847971-provider-credentials\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:28.120858 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:28.120833    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b9f9140-739a-4f76-9165-6af845847971-plugins-config-volume" pod "1b9f9140-739a-4f76-9165-6af845847971" (UID: "1b9f9140-739a-4f76-9165-6af845847971"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:28:28.121644 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:28.121619    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b9f9140-739a-4f76-9165-6af845847971-routing-overlay-volume" pod "1b9f9140-739a-4f76-9165-6af845847971" (UID: "1b9f9140-739a-4f76-9165-6af845847971"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:28:28.124629 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:28.124602    5591 scope.go:122] "RemoveContainer" containerID="a16e5529c29277947c6f685c5b2e0ea1300314b9dfabeeb54f6ec4fc19c8724c"
Nov 03 20:28:28.124950 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:28:28.124925    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a16e5529c29277947c6f685c5b2e0ea1300314b9dfabeeb54f6ec4fc19c8724c\": container with ID starting with a16e5529c29277947c6f685c5b2e0ea1300314b9dfabeeb54f6ec4fc19c8724c not found: ID does not exist" containerID="a16e5529c29277947c6f685c5b2e0ea1300314b9dfabeeb54f6ec4fc19c8724c"
Nov 03 20:28:28.125046 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:28.124962    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a16e5529c29277947c6f685c5b2e0ea1300314b9dfabeeb54f6ec4fc19c8724c"} err="failed to get container status \"a16e5529c29277947c6f685c5b2e0ea1300314b9dfabeeb54f6ec4fc19c8724c\": rpc error: code = NotFound desc = could not find container \"a16e5529c29277947c6f685c5b2e0ea1300314b9dfabeeb54f6ec4fc19c8724c\": container with ID starting with a16e5529c29277947c6f685c5b2e0ea1300314b9dfabeeb54f6ec4fc19c8724c not found: ID does not exist"
Nov 03 20:28:28.222077 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:28.221989    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b9f9140-739a-4f76-9165-6af845847971-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:28.222077 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:28.222025    5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b9f9140-739a-4f76-9165-6af845847971-routing-overlay-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:28.439630 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:28.439597    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf"]
Nov 03 20:28:28.443912 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:28.443887    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8b7351/payload-processing-external-model-e2e-extmod-a-8b7351-7695gcpsf"]
Nov 03 20:28:28.786714 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:28:28.786680    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:28:29.353561 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:29.353529    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b9f9140-739a-4f76-9165-6af845847971" path="/var/lib/kubelet/pods/1b9f9140-739a-4f76-9165-6af845847971/volumes"
Nov 03 20:28:30.260851 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:30.260807    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45"]
Nov 03 20:28:35.261240 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.261150    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45" podUID="5e92bb59-2a3f-4286-90eb-d33f8f857c9d" containerName="payload-processing" containerID="cri-o://68f2ba4ae81a9a43aa31ffbaf6a1a571f81d264ef36971f7b22c9fcde2bb727e" gracePeriod=25
Nov 03 20:28:35.587902 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.587873    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45"
Nov 03 20:28:35.621052 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.621008    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"]
Nov 03 20:28:35.622520 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.622491    5591 cpu_manager.go:414] "containerMap: removing container" podUID="1b9f9140-739a-4f76-9165-6af845847971" containerName="payload-processing"
Nov 03 20:28:35.622520 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.622519    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b9f9140-739a-4f76-9165-6af845847971" containerName="payload-processing"
Nov 03 20:28:35.622743 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.622555    5591 cpu_manager.go:414] "containerMap: removing container" podUID="5e92bb59-2a3f-4286-90eb-d33f8f857c9d" containerName="payload-processing"
Nov 03 20:28:35.622743 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.622563    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e92bb59-2a3f-4286-90eb-d33f8f857c9d" containerName="payload-processing"
Nov 03 20:28:35.622861 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.622795    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b9f9140-739a-4f76-9165-6af845847971" containerName="payload-processing"
Nov 03 20:28:35.622861 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.622820    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e92bb59-2a3f-4286-90eb-d33f8f857c9d" containerName="payload-processing"
Nov 03 20:28:35.627679 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.627643    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.633920 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.633872    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-086365-openshift-default-dockercfg-72qdc\""
Nov 03 20:28:35.644743 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.644707    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"]
Nov 03 20:28:35.672945 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.672902    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e92bb59-2a3f-4286-90eb-d33f8f857c9d-kube-api-access-blvqr\" (UniqueName: \"kubernetes.io/projected/5e92bb59-2a3f-4286-90eb-d33f8f857c9d-kube-api-access-blvqr\") pod \"5e92bb59-2a3f-4286-90eb-d33f8f857c9d\" (UID: \"5e92bb59-2a3f-4286-90eb-d33f8f857c9d\") "
Nov 03 20:28:35.673235 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.673192    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e92bb59-2a3f-4286-90eb-d33f8f857c9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e92bb59-2a3f-4286-90eb-d33f8f857c9d-plugins-config-volume\") pod \"5e92bb59-2a3f-4286-90eb-d33f8f857c9d\" (UID: \"5e92bb59-2a3f-4286-90eb-d33f8f857c9d\") "
Nov 03 20:28:35.673563 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.673535    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e92bb59-2a3f-4286-90eb-d33f8f857c9d-plugins-config-volume" pod "5e92bb59-2a3f-4286-90eb-d33f8f857c9d" (UID: "5e92bb59-2a3f-4286-90eb-d33f8f857c9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:28:35.674314 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.674288    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e92bb59-2a3f-4286-90eb-d33f8f857c9d-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:35.675643 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.675619    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e92bb59-2a3f-4286-90eb-d33f8f857c9d-kube-api-access-blvqr" pod "5e92bb59-2a3f-4286-90eb-d33f8f857c9d" (UID: "5e92bb59-2a3f-4286-90eb-d33f8f857c9d"). InnerVolumeSpecName "kube-api-access-blvqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:28:35.776074 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.776019    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-workload-socket\") pod \"e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") " pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.776074 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.776077    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d381317-b122-4b60-989e-44e600b1c6eb-istio-podinfo\") pod \"e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") " pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.776413 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.776106    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d381317-b122-4b60-989e-44e600b1c6eb-istio-token\") pod \"e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") " pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.776413 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.776298    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-istio-data\") pod \"e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") " pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.776555 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.776534    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d381317-b122-4b60-989e-44e600b1c6eb-istiod-ca-cert\") pod \"e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") " pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.776705 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.776675    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4nnc\" (UniqueName: \"kubernetes.io/projected/3d381317-b122-4b60-989e-44e600b1c6eb-kube-api-access-b4nnc\") pod \"e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") " pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.776784 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.776728    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-credential-socket\") pod \"e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") " pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.776829 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.776816    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-workload-certs\") pod \"e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") " pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.776869 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.776842    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-istio-envoy\") pod \"e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") " pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.776972 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.776957    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blvqr\" (UniqueName: \"kubernetes.io/projected/5e92bb59-2a3f-4286-90eb-d33f8f857c9d-kube-api-access-blvqr\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:28:35.878940 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.878850    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d381317-b122-4b60-989e-44e600b1c6eb-istiod-ca-cert\") pod \"e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") " pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.879118 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.878951    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4nnc\" (UniqueName: \"kubernetes.io/projected/3d381317-b122-4b60-989e-44e600b1c6eb-kube-api-access-b4nnc\") pod \"e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") " pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.879118 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.878986    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-credential-socket\") pod \"e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") " pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.879118 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.879020    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-workload-certs\") pod \"e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") " pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.879118 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.879050    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-istio-envoy\") pod \"e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") " pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.879395 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.879122    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-workload-socket\") pod \"e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") " pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.879395 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.879146    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d381317-b122-4b60-989e-44e600b1c6eb-istio-podinfo\") pod \"e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") " pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.879395 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.879191    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d381317-b122-4b60-989e-44e600b1c6eb-istio-token\") pod \"e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") " pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.879395 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.879251    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-istio-data\") pod \"e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") " pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.879626 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.879409    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-credential-socket\") pod \"e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") " pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.879626 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.879576    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-workload-certs\") pod \"e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") " pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.879748 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.879665    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-workload-socket\") pod \"e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") " pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.879819 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.879794    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d381317-b122-4b60-989e-44e600b1c6eb-istiod-ca-cert\") pod \"e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") " pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.879819 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.879801    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-istio-data\") pod \"e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") " pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.882099 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.882072    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d381317-b122-4b60-989e-44e600b1c6eb-istio-podinfo\") pod \"e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") " pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.882099 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.882093    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-istio-envoy\") pod \"e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") " pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.891557 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.891526    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d381317-b122-4b60-989e-44e600b1c6eb-istio-token\") pod \"e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") " pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.891706 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.891595    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4nnc\" (UniqueName: \"kubernetes.io/projected/3d381317-b122-4b60-989e-44e600b1c6eb-kube-api-access-b4nnc\") pod \"e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") " pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:35.965238 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:35.962565    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:36.240855 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:36.240772    5591 generic.go:356] "Generic (PLEG): container finished" podID="5e92bb59-2a3f-4286-90eb-d33f8f857c9d" containerID="68f2ba4ae81a9a43aa31ffbaf6a1a571f81d264ef36971f7b22c9fcde2bb727e" exitCode=0
Nov 03 20:28:36.241019 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:36.240850    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45"
Nov 03 20:28:36.241019 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:36.240846    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45" event={"ID":"5e92bb59-2a3f-4286-90eb-d33f8f857c9d","Type":"ContainerDied","Data":"68f2ba4ae81a9a43aa31ffbaf6a1a571f81d264ef36971f7b22c9fcde2bb727e"}
Nov 03 20:28:36.241019 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:36.240999    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45" event={"ID":"5e92bb59-2a3f-4286-90eb-d33f8f857c9d","Type":"ContainerDied","Data":"81b9f79e56fb25678648008876cf5680c5eff2ea76b70fd19c475017fd6bfe7a"}
Nov 03 20:28:36.241167 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:36.241021    5591 scope.go:122] "RemoveContainer" containerID="68f2ba4ae81a9a43aa31ffbaf6a1a571f81d264ef36971f7b22c9fcde2bb727e"
Nov 03 20:28:36.253719 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:36.253700    5591 scope.go:122] "RemoveContainer" containerID="68f2ba4ae81a9a43aa31ffbaf6a1a571f81d264ef36971f7b22c9fcde2bb727e"
Nov 03 20:28:36.254039 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:28:36.254012    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68f2ba4ae81a9a43aa31ffbaf6a1a571f81d264ef36971f7b22c9fcde2bb727e\": container with ID starting with 68f2ba4ae81a9a43aa31ffbaf6a1a571f81d264ef36971f7b22c9fcde2bb727e not found: ID does not exist" containerID="68f2ba4ae81a9a43aa31ffbaf6a1a571f81d264ef36971f7b22c9fcde2bb727e"
Nov 03 20:28:36.254125 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:36.254051    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68f2ba4ae81a9a43aa31ffbaf6a1a571f81d264ef36971f7b22c9fcde2bb727e"} err="failed to get container status \"68f2ba4ae81a9a43aa31ffbaf6a1a571f81d264ef36971f7b22c9fcde2bb727e\": rpc error: code = NotFound desc = could not find container \"68f2ba4ae81a9a43aa31ffbaf6a1a571f81d264ef36971f7b22c9fcde2bb727e\": container with ID starting with 68f2ba4ae81a9a43aa31ffbaf6a1a571f81d264ef36971f7b22c9fcde2bb727e not found: ID does not exist"
Nov 03 20:28:36.267809 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:36.267785    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45"]
Nov 03 20:28:36.271616 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:28:36.271573    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d381317_b122_4b60_989e_44e600b1c6eb.slice/crio-078c38fe6ef1f7f46f8e7ff3cc80a3dc97e2e80fb13cf278b3090478a6b8682f WatchSource:0}: Error finding container 078c38fe6ef1f7f46f8e7ff3cc80a3dc97e2e80fb13cf278b3090478a6b8682f: Status 404 returned error can't find the container with id 078c38fe6ef1f7f46f8e7ff3cc80a3dc97e2e80fb13cf278b3090478a6b8682f
Nov 03 20:28:36.273105 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:36.273083    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"]
Nov 03 20:28:36.274910 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:36.274875    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:28:36.275013 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:36.274945    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:28:36.275013 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:36.274977    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:28:36.278050 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:36.278019    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d2aa3b-b894cb6db-7kg45"]
Nov 03 20:28:37.252659 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:37.252618    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg" event={"ID":"3d381317-b122-4b60-989e-44e600b1c6eb","Type":"ContainerStarted","Data":"edb4b320110bc96981e76a05937963cb67eb3427f69ee0843d19186ea09dbf90"}
Nov 03 20:28:37.252871 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:37.252694    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg" event={"ID":"3d381317-b122-4b60-989e-44e600b1c6eb","Type":"ContainerStarted","Data":"078c38fe6ef1f7f46f8e7ff3cc80a3dc97e2e80fb13cf278b3090478a6b8682f"}
Nov 03 20:28:37.280507 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:37.280453    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg" podStartSLOduration=7.280436979 podStartE2EDuration="7.280436979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:37.277884697 +0000 UTC m=+3066.050177707" watchObservedRunningTime="2026-11-03 20:28:37.280436979 +0000 UTC m=+3066.052729996"
Nov 03 20:28:37.352698 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:37.352663    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e92bb59-2a3f-4286-90eb-d33f8f857c9d" path="/var/lib/kubelet/pods/5e92bb59-2a3f-4286-90eb-d33f8f857c9d/volumes"
Nov 03 20:28:37.963832 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:37.963796    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:37.970790 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:37.970761    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:38.260932 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:38.260849    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:38.262095 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:28:38.262077    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:28:58.739508 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:28:58.739465    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:29:02.363422 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:02.363387    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-bfmpt"]
Nov 03 20:29:02.368596 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:02.368574    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bfmpt"
Nov 03 20:29:02.381912 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:02.381641    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 20:29:02.381912 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:02.381690    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4p6h4\""
Nov 03 20:29:02.381912 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:02.381736    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 20:29:02.384753 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:02.384728    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-bfmpt"]
Nov 03 20:29:02.474884 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:02.474845    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g65wf\" (UniqueName: \"kubernetes.io/projected/7353c7e4-f98e-4669-a604-22615755772f-kube-api-access-g65wf\") pod \"migrator-66df5d496b-bfmpt\" (UID: \"7353c7e4-f98e-4669-a604-22615755772f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bfmpt"
Nov 03 20:29:02.618319 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:02.617126    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g65wf\" (UniqueName: \"kubernetes.io/projected/7353c7e4-f98e-4669-a604-22615755772f-kube-api-access-g65wf\") pod \"migrator-66df5d496b-bfmpt\" (UID: \"7353c7e4-f98e-4669-a604-22615755772f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bfmpt"
Nov 03 20:29:02.631951 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:02.631910    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g65wf\" (UniqueName: \"kubernetes.io/projected/7353c7e4-f98e-4669-a604-22615755772f-kube-api-access-g65wf\") pod \"migrator-66df5d496b-bfmpt\" (UID: \"7353c7e4-f98e-4669-a604-22615755772f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bfmpt"
Nov 03 20:29:02.685402 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:02.685367    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bfmpt"
Nov 03 20:29:02.980110 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:29:02.980077    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7353c7e4_f98e_4669_a604_22615755772f.slice/crio-39d31ee86fac47c31a41f8b46393b6d7bde0bfe804a6278547b7cfe41b682e8b WatchSource:0}: Error finding container 39d31ee86fac47c31a41f8b46393b6d7bde0bfe804a6278547b7cfe41b682e8b: Status 404 returned error can't find the container with id 39d31ee86fac47c31a41f8b46393b6d7bde0bfe804a6278547b7cfe41b682e8b
Nov 03 20:29:02.980369 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:02.980343    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-bfmpt"]
Nov 03 20:29:02.982277 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:02.982260    5591 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:29:03.678539 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:03.678503    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bfmpt" event={"ID":"7353c7e4-f98e-4669-a604-22615755772f","Type":"ContainerStarted","Data":"39d31ee86fac47c31a41f8b46393b6d7bde0bfe804a6278547b7cfe41b682e8b"}
Nov 03 20:29:04.692271 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:04.692168    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bfmpt" event={"ID":"7353c7e4-f98e-4669-a604-22615755772f","Type":"ContainerStarted","Data":"c7341566d5c074b8bdb7677ca4aeb099bf1491d846718a8e62e056fb854fb348"}
Nov 03 20:29:04.710926 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:04.710874    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bfmpt" podStartSLOduration=1.159659842 podStartE2EDuration="2.710858129s" totalImagesPullingTime="1.551198287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:04.710027167 +0000 UTC m=+3093.482320186" watchObservedRunningTime="2026-11-03 20:29:04.710858129 +0000 UTC m=+3093.483151180"
Nov 03 20:29:28.727784 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:29:28.727740    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:29:39.649835 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:39.649799    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"]
Nov 03 20:29:39.650554 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:39.650057    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg" podUID="3d381317-b122-4b60-989e-44e600b1c6eb" containerName="istio-proxy" containerID="cri-o://edb4b320110bc96981e76a05937963cb67eb3427f69ee0843d19186ea09dbf90" gracePeriod=30
Nov 03 20:29:44.955252 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:44.955224    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:29:44.993334 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:44.993301    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"]
Nov 03 20:29:44.994308 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:44.994292    5591 cpu_manager.go:414] "containerMap: removing container" podUID="3d381317-b122-4b60-989e-44e600b1c6eb" containerName="istio-proxy"
Nov 03 20:29:44.994363 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:44.994310    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d381317-b122-4b60-989e-44e600b1c6eb" containerName="istio-proxy"
Nov 03 20:29:44.994520 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:44.994509    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d381317-b122-4b60-989e-44e600b1c6eb" containerName="istio-proxy"
Nov 03 20:29:44.999054 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:44.999033    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.004580 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.004540    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-96230c5b-openshift-default-dockercfg-tb8k7\""
Nov 03 20:29:45.022722 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.022690    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-istio-data\") pod \"3d381317-b122-4b60-989e-44e600b1c6eb\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") "
Nov 03 20:29:45.023013 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.022806    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-credential-socket\") pod \"3d381317-b122-4b60-989e-44e600b1c6eb\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") "
Nov 03 20:29:45.023106 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.023024    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-istio-data" pod "3d381317-b122-4b60-989e-44e600b1c6eb" (UID: "3d381317-b122-4b60-989e-44e600b1c6eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:45.023409 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.023376    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"]
Nov 03 20:29:45.023515 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.023376    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-workload-socket" pod "3d381317-b122-4b60-989e-44e600b1c6eb" (UID: "3d381317-b122-4b60-989e-44e600b1c6eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:45.026333 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.026303    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-credential-socket" pod "3d381317-b122-4b60-989e-44e600b1c6eb" (UID: "3d381317-b122-4b60-989e-44e600b1c6eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:45.026607 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.026585    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-workload-socket\") pod \"3d381317-b122-4b60-989e-44e600b1c6eb\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") "
Nov 03 20:29:45.026699 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.026661    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d381317-b122-4b60-989e-44e600b1c6eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d381317-b122-4b60-989e-44e600b1c6eb-istio-podinfo\") pod \"3d381317-b122-4b60-989e-44e600b1c6eb\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") "
Nov 03 20:29:45.026767 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.026714    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d381317-b122-4b60-989e-44e600b1c6eb-kube-api-access-b4nnc\" (UniqueName: \"kubernetes.io/projected/3d381317-b122-4b60-989e-44e600b1c6eb-kube-api-access-b4nnc\") pod \"3d381317-b122-4b60-989e-44e600b1c6eb\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") "
Nov 03 20:29:45.026838 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.026809    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d381317-b122-4b60-989e-44e600b1c6eb-istio-token\" (UniqueName: \"kubernetes.io/projected/3d381317-b122-4b60-989e-44e600b1c6eb-istio-token\") pod \"3d381317-b122-4b60-989e-44e600b1c6eb\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") "
Nov 03 20:29:45.026924 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.026908    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d381317-b122-4b60-989e-44e600b1c6eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d381317-b122-4b60-989e-44e600b1c6eb-istiod-ca-cert\") pod \"3d381317-b122-4b60-989e-44e600b1c6eb\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") "
Nov 03 20:29:45.026992 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.026937    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-istio-envoy\") pod \"3d381317-b122-4b60-989e-44e600b1c6eb\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") "
Nov 03 20:29:45.027055 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.027046    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-workload-certs\") pod \"3d381317-b122-4b60-989e-44e600b1c6eb\" (UID: \"3d381317-b122-4b60-989e-44e600b1c6eb\") "
Nov 03 20:29:45.028055 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.028028    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:45.028172 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.028057    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:45.028172 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.028072    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:45.028360 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.028337    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-workload-certs" pod "3d381317-b122-4b60-989e-44e600b1c6eb" (UID: "3d381317-b122-4b60-989e-44e600b1c6eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:45.028664 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.028643    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d381317-b122-4b60-989e-44e600b1c6eb-istiod-ca-cert" pod "3d381317-b122-4b60-989e-44e600b1c6eb" (UID: "3d381317-b122-4b60-989e-44e600b1c6eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:29:45.029801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.029775    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d381317-b122-4b60-989e-44e600b1c6eb-istio-podinfo" pod "3d381317-b122-4b60-989e-44e600b1c6eb" (UID: "3d381317-b122-4b60-989e-44e600b1c6eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:29:45.030482 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.030455    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d381317-b122-4b60-989e-44e600b1c6eb-istio-token" pod "3d381317-b122-4b60-989e-44e600b1c6eb" (UID: "3d381317-b122-4b60-989e-44e600b1c6eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:45.032363 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.032338    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-istio-envoy" pod "3d381317-b122-4b60-989e-44e600b1c6eb" (UID: "3d381317-b122-4b60-989e-44e600b1c6eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:29:45.032456 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.032370    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d381317-b122-4b60-989e-44e600b1c6eb-kube-api-access-b4nnc" pod "3d381317-b122-4b60-989e-44e600b1c6eb" (UID: "3d381317-b122-4b60-989e-44e600b1c6eb"). InnerVolumeSpecName "kube-api-access-b4nnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:29:45.129599 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.129561    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m29z4\" (UniqueName: \"kubernetes.io/projected/e8286c74-9fa8-4b61-ac75-7b758308cd13-kube-api-access-m29z4\") pod \"e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") " pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.129799 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.129629    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-data\") pod \"e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") " pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.129799 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.129767    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-podinfo\") pod \"e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") " pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.129901 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.129870    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-token\") pod \"e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") " pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.129945 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.129920    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-workload-certs\") pod \"e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") " pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.130004 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.129956    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-credential-socket\") pod \"e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") " pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.130057 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.130046    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8286c74-9fa8-4b61-ac75-7b758308cd13-istiod-ca-cert\") pod \"e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") " pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.130119 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.130104    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-workload-socket\") pod \"e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") " pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.130171 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.130149    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-envoy\") pod \"e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") " pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.130435 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.130411    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:45.130483 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.130444    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d381317-b122-4b60-989e-44e600b1c6eb-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:45.130483 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.130461    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4nnc\" (UniqueName: \"kubernetes.io/projected/3d381317-b122-4b60-989e-44e600b1c6eb-kube-api-access-b4nnc\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:45.130483 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.130477    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d381317-b122-4b60-989e-44e600b1c6eb-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:45.130581 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.130492    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d381317-b122-4b60-989e-44e600b1c6eb-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:45.130581 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.130505    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d381317-b122-4b60-989e-44e600b1c6eb-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:29:45.211042 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.210960    5591 generic.go:356] "Generic (PLEG): container finished" podID="3d381317-b122-4b60-989e-44e600b1c6eb" containerID="edb4b320110bc96981e76a05937963cb67eb3427f69ee0843d19186ea09dbf90" exitCode=0
Nov 03 20:29:45.211042 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.211021    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"
Nov 03 20:29:45.211290 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.211058    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg" event={"ID":"3d381317-b122-4b60-989e-44e600b1c6eb","Type":"ContainerDied","Data":"edb4b320110bc96981e76a05937963cb67eb3427f69ee0843d19186ea09dbf90"}
Nov 03 20:29:45.211290 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.211111    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg" event={"ID":"3d381317-b122-4b60-989e-44e600b1c6eb","Type":"ContainerDied","Data":"078c38fe6ef1f7f46f8e7ff3cc80a3dc97e2e80fb13cf278b3090478a6b8682f"}
Nov 03 20:29:45.211290 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.211134    5591 scope.go:122] "RemoveContainer" containerID="edb4b320110bc96981e76a05937963cb67eb3427f69ee0843d19186ea09dbf90"
Nov 03 20:29:45.230751 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.230731    5591 scope.go:122] "RemoveContainer" containerID="edb4b320110bc96981e76a05937963cb67eb3427f69ee0843d19186ea09dbf90"
Nov 03 20:29:45.231118 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:29:45.231086    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb4b320110bc96981e76a05937963cb67eb3427f69ee0843d19186ea09dbf90\": container with ID starting with edb4b320110bc96981e76a05937963cb67eb3427f69ee0843d19186ea09dbf90 not found: ID does not exist" containerID="edb4b320110bc96981e76a05937963cb67eb3427f69ee0843d19186ea09dbf90"
Nov 03 20:29:45.231274 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.231127    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb4b320110bc96981e76a05937963cb67eb3427f69ee0843d19186ea09dbf90"} err="failed to get container status \"edb4b320110bc96981e76a05937963cb67eb3427f69ee0843d19186ea09dbf90\": rpc error: code = NotFound desc = could not find container \"edb4b320110bc96981e76a05937963cb67eb3427f69ee0843d19186ea09dbf90\": container with ID starting with edb4b320110bc96981e76a05937963cb67eb3427f69ee0843d19186ea09dbf90 not found: ID does not exist"
Nov 03 20:29:45.231667 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.231648    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m29z4\" (UniqueName: \"kubernetes.io/projected/e8286c74-9fa8-4b61-ac75-7b758308cd13-kube-api-access-m29z4\") pod \"e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") " pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.231713 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.231694    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-data\") pod \"e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") " pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.231752 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.231730    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-podinfo\") pod \"e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") " pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.231801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.231790    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-token\") pod \"e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") " pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.231920 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.231837    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-workload-certs\") pod \"e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") " pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.231920 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.231877    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-credential-socket\") pod \"e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") " pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.232045 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.231919    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8286c74-9fa8-4b61-ac75-7b758308cd13-istiod-ca-cert\") pod \"e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") " pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.232045 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.231993    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-workload-socket\") pod \"e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") " pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.232535 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.232054    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-envoy\") pod \"e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") " pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.232535 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.232237    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-data\") pod \"e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") " pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.232535 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.232287    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-workload-certs\") pod \"e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") " pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.232535 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.232319    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-credential-socket\") pod \"e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") " pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.232535 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.232394    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-workload-socket\") pod \"e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") " pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.232981 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.232953    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8286c74-9fa8-4b61-ac75-7b758308cd13-istiod-ca-cert\") pod \"e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") " pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.234476 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.234461    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-envoy\") pod \"e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") " pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.234531 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.234510    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-podinfo\") pod \"e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") " pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.243841 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.243810    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"]
Nov 03 20:29:45.249572 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.249550    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m29z4\" (UniqueName: \"kubernetes.io/projected/e8286c74-9fa8-4b61-ac75-7b758308cd13-kube-api-access-m29z4\") pod \"e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") " pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.249690 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.249602    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-086365-openshift-default-65b8db699f-jqqcg"]
Nov 03 20:29:45.250105 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.250087    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-token\") pod \"e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") " pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.323943 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.323902    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:45.374902 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.374872    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d381317-b122-4b60-989e-44e600b1c6eb" path="/var/lib/kubelet/pods/3d381317-b122-4b60-989e-44e600b1c6eb/volumes"
Nov 03 20:29:45.655181 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:29:45.655149    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8286c74_9fa8_4b61_ac75_7b758308cd13.slice/crio-d98a4ac19827e169719a419ce34fa2381e584cb506e6dc4c2d14a16c608311a6 WatchSource:0}: Error finding container d98a4ac19827e169719a419ce34fa2381e584cb506e6dc4c2d14a16c608311a6: Status 404 returned error can't find the container with id d98a4ac19827e169719a419ce34fa2381e584cb506e6dc4c2d14a16c608311a6
Nov 03 20:29:45.655534 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.655510    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"]
Nov 03 20:29:45.658392 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.658356    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:29:45.658511 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.658436    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:29:45.658511 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:45.658477    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:29:46.221291 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:46.221249    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm" event={"ID":"e8286c74-9fa8-4b61-ac75-7b758308cd13","Type":"ContainerStarted","Data":"c6e580505a0c66338d968c496ed003a03c754f43ab1b74f201e79c116b788dff"}
Nov 03 20:29:46.221747 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:46.221357    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm" event={"ID":"e8286c74-9fa8-4b61-ac75-7b758308cd13","Type":"ContainerStarted","Data":"d98a4ac19827e169719a419ce34fa2381e584cb506e6dc4c2d14a16c608311a6"}
Nov 03 20:29:46.274227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:46.273693    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm" podStartSLOduration=39.273672104 podStartE2EDuration="39.273672104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:46.250707112 +0000 UTC m=+3135.023000184" watchObservedRunningTime="2026-11-03 20:29:46.273672104 +0000 UTC m=+3135.045965123"
Nov 03 20:29:46.325077 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:46.325043    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:46.331499 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:46.331468    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:47.230357 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:47.230325    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:47.231439 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:29:47.231419    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:29:58.733408 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:29:58.733364    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:30:23.516232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:23.515496    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:23.610484 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:23.610430    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:23.705380 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:23.705324    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded"
Nov 03 20:30:23.821485 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:23.821371    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:23.925235 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:23.899009    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:23.925235 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:23.918385    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:24.296237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:24.295349    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl" podUID="97472d8f-eb02-4974-9a3d-275a65ca555b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.247:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:28.728045 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:30:28.728005    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:30:42.566719 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:42.566679    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"]
Nov 03 20:30:57.309343 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:57.309295    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl" podUID="97472d8f-eb02-4974-9a3d-275a65ca555b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.247:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:57.424487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:57.424424    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87" podUID="666c0175-cb77-4af6-8793-81a7a684c1f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.249:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:57.555558 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:57.555509    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:57.755449 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:57.738781    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl" podUID="97472d8f-eb02-4974-9a3d-275a65ca555b" containerName="main" containerID="cri-o://ea308f815a7c96ece2e89e4415f6eafbf41e1edb6cd91755aaf8fad8651086e8" gracePeriod=45
Nov 03 20:30:57.813345 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:57.813300    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 20:30:58.200370 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:58.198755    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl" podUID="97472d8f-eb02-4974-9a3d-275a65ca555b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.247:8000/ready\": read tcp 10.134.0.2:47460->10.134.0.247:8000: read: connection reset by peer"
Nov 03 20:30:58.200370 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:58.198911    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"
Nov 03 20:30:58.272694 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:58.272377    5591 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87" podUID="666c0175-cb77-4af6-8793-81a7a684c1f2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.249:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:30:58.291460 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:58.291124    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl" podUID="97472d8f-eb02-4974-9a3d-275a65ca555b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.247:8000/ready\": dial tcp 10.134.0.247:8000: connect: connection refused"
Nov 03 20:30:58.519508 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:58.519473    5591 generic.go:356] "Generic (PLEG): container finished" podID="97472d8f-eb02-4974-9a3d-275a65ca555b" containerID="ea308f815a7c96ece2e89e4415f6eafbf41e1edb6cd91755aaf8fad8651086e8" exitCode=0
Nov 03 20:30:58.520040 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:58.520020    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl" event={"ID":"97472d8f-eb02-4974-9a3d-275a65ca555b","Type":"ContainerDied","Data":"ea308f815a7c96ece2e89e4415f6eafbf41e1edb6cd91755aaf8fad8651086e8"}
Nov 03 20:30:58.714352 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:58.714316    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"
Nov 03 20:30:58.736972 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:30:58.736928    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:30:58.815539 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:58.815432    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-dshm\") pod \"97472d8f-eb02-4974-9a3d-275a65ca555b\" (UID: \"97472d8f-eb02-4974-9a3d-275a65ca555b\") "
Nov 03 20:30:58.815539 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:58.815483    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-home\" (UniqueName: \"kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-home\") pod \"97472d8f-eb02-4974-9a3d-275a65ca555b\" (UID: \"97472d8f-eb02-4974-9a3d-275a65ca555b\") "
Nov 03 20:30:58.815787 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:58.815635    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-tmp-dir\") pod \"97472d8f-eb02-4974-9a3d-275a65ca555b\" (UID: \"97472d8f-eb02-4974-9a3d-275a65ca555b\") "
Nov 03 20:30:58.815787 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:58.815777    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-model-cache\") pod \"97472d8f-eb02-4974-9a3d-275a65ca555b\" (UID: \"97472d8f-eb02-4974-9a3d-275a65ca555b\") "
Nov 03 20:30:58.815910 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:58.815807    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-home" pod "97472d8f-eb02-4974-9a3d-275a65ca555b" (UID: "97472d8f-eb02-4974-9a3d-275a65ca555b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:30:58.815964 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:58.815909    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-tmp-dir" pod "97472d8f-eb02-4974-9a3d-275a65ca555b" (UID: "97472d8f-eb02-4974-9a3d-275a65ca555b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:30:58.815964 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:58.815923    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97472d8f-eb02-4974-9a3d-275a65ca555b-tls-certs\" (UniqueName: \"kubernetes.io/secret/97472d8f-eb02-4974-9a3d-275a65ca555b-tls-certs\") pod \"97472d8f-eb02-4974-9a3d-275a65ca555b\" (UID: \"97472d8f-eb02-4974-9a3d-275a65ca555b\") "
Nov 03 20:30:58.815964 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:58.815961    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97472d8f-eb02-4974-9a3d-275a65ca555b-kube-api-access-t8pnq\" (UniqueName: \"kubernetes.io/projected/97472d8f-eb02-4974-9a3d-275a65ca555b-kube-api-access-t8pnq\") pod \"97472d8f-eb02-4974-9a3d-275a65ca555b\" (UID: \"97472d8f-eb02-4974-9a3d-275a65ca555b\") "
Nov 03 20:30:58.816098 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:58.816056    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-model-cache" pod "97472d8f-eb02-4974-9a3d-275a65ca555b" (UID: "97472d8f-eb02-4974-9a3d-275a65ca555b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:30:58.817143 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:58.817113    5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-home\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:58.817337 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:58.817146    5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-tmp-dir\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:58.817337 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:58.817161    5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-model-cache\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:58.818973 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:58.818937    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-dshm" pod "97472d8f-eb02-4974-9a3d-275a65ca555b" (UID: "97472d8f-eb02-4974-9a3d-275a65ca555b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:30:58.819091 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:58.819061    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97472d8f-eb02-4974-9a3d-275a65ca555b-tls-certs" pod "97472d8f-eb02-4974-9a3d-275a65ca555b" (UID: "97472d8f-eb02-4974-9a3d-275a65ca555b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:30:58.825398 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:58.825362    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97472d8f-eb02-4974-9a3d-275a65ca555b-kube-api-access-t8pnq" pod "97472d8f-eb02-4974-9a3d-275a65ca555b" (UID: "97472d8f-eb02-4974-9a3d-275a65ca555b"). InnerVolumeSpecName "kube-api-access-t8pnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:30:58.919431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:58.919394    5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97472d8f-eb02-4974-9a3d-275a65ca555b-tls-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:58.919431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:58.919429    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8pnq\" (UniqueName: \"kubernetes.io/projected/97472d8f-eb02-4974-9a3d-275a65ca555b-kube-api-access-t8pnq\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:58.919431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:58.919440    5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97472d8f-eb02-4974-9a3d-275a65ca555b-dshm\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:30:59.530957 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:59.530918    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl" event={"ID":"97472d8f-eb02-4974-9a3d-275a65ca555b","Type":"ContainerDied","Data":"ace566ecd9b1e3a6eade0ee5726fca71a4b08f0bbff444af43f5da5d9596df7a"}
Nov 03 20:30:59.531530 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:59.530982    5591 scope.go:122] "RemoveContainer" containerID="ea308f815a7c96ece2e89e4415f6eafbf41e1edb6cd91755aaf8fad8651086e8"
Nov 03 20:30:59.531530 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:59.530932    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"
Nov 03 20:30:59.556514 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:59.556476    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"]
Nov 03 20:30:59.561687 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:30:59.561652    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e7b70d/test-model-e7b70d-kserve-677dbc94c7-zk4kl"]
Nov 03 20:31:01.353698 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:01.353661    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97472d8f-eb02-4974-9a3d-275a65ca555b" path="/var/lib/kubelet/pods/97472d8f-eb02-4974-9a3d-275a65ca555b/volumes"
Nov 03 20:31:01.426282 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:01.426240    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"]
Nov 03 20:31:01.426532 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:01.426506    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm" podUID="e8286c74-9fa8-4b61-ac75-7b758308cd13" containerName="istio-proxy" containerID="cri-o://c6e580505a0c66338d968c496ed003a03c754f43ab1b74f201e79c116b788dff" gracePeriod=30
Nov 03 20:31:02.316642 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:02.314444    5591 patch_prober.go:29] interesting pod/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:31:02.316642 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:02.314515    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm" podUID="e8286c74-9fa8-4b61-ac75-7b758308cd13" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:31:02.916267 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:02.916227    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"]
Nov 03 20:31:06.685528 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.685244    5591 generic.go:356] "Generic (PLEG): container finished" podID="e8286c74-9fa8-4b61-ac75-7b758308cd13" containerID="c6e580505a0c66338d968c496ed003a03c754f43ab1b74f201e79c116b788dff" exitCode=0
Nov 03 20:31:06.685528 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.685364    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm" event={"ID":"e8286c74-9fa8-4b61-ac75-7b758308cd13","Type":"ContainerDied","Data":"c6e580505a0c66338d968c496ed003a03c754f43ab1b74f201e79c116b788dff"}
Nov 03 20:31:06.747757 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.747722    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:31:06.837475 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.837436    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-credential-socket\") pod \"e8286c74-9fa8-4b61-ac75-7b758308cd13\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") "
Nov 03 20:31:06.837732 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.837564    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-envoy\") pod \"e8286c74-9fa8-4b61-ac75-7b758308cd13\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") "
Nov 03 20:31:06.837732 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.837672    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-podinfo\") pod \"e8286c74-9fa8-4b61-ac75-7b758308cd13\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") "
Nov 03 20:31:06.837881 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.837760    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8286c74-9fa8-4b61-ac75-7b758308cd13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8286c74-9fa8-4b61-ac75-7b758308cd13-istiod-ca-cert\") pod \"e8286c74-9fa8-4b61-ac75-7b758308cd13\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") "
Nov 03 20:31:06.837881 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.837810    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-data\") pod \"e8286c74-9fa8-4b61-ac75-7b758308cd13\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") "
Nov 03 20:31:06.838008 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.837905    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-token\" (UniqueName: \"kubernetes.io/projected/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-token\") pod \"e8286c74-9fa8-4b61-ac75-7b758308cd13\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") "
Nov 03 20:31:06.838256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.838095    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-workload-socket\") pod \"e8286c74-9fa8-4b61-ac75-7b758308cd13\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") "
Nov 03 20:31:06.838256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.838135    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-workload-certs\") pod \"e8286c74-9fa8-4b61-ac75-7b758308cd13\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") "
Nov 03 20:31:06.838256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.838185    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8286c74-9fa8-4b61-ac75-7b758308cd13-kube-api-access-m29z4\" (UniqueName: \"kubernetes.io/projected/e8286c74-9fa8-4b61-ac75-7b758308cd13-kube-api-access-m29z4\") pod \"e8286c74-9fa8-4b61-ac75-7b758308cd13\" (UID: \"e8286c74-9fa8-4b61-ac75-7b758308cd13\") "
Nov 03 20:31:06.838962 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.838901    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8286c74-9fa8-4b61-ac75-7b758308cd13-istiod-ca-cert" pod "e8286c74-9fa8-4b61-ac75-7b758308cd13" (UID: "e8286c74-9fa8-4b61-ac75-7b758308cd13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:06.839221 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.839086    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-workload-socket" pod "e8286c74-9fa8-4b61-ac75-7b758308cd13" (UID: "e8286c74-9fa8-4b61-ac75-7b758308cd13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:06.839333 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.839315    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-workload-certs" pod "e8286c74-9fa8-4b61-ac75-7b758308cd13" (UID: "e8286c74-9fa8-4b61-ac75-7b758308cd13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:06.839734 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.839500    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-data" pod "e8286c74-9fa8-4b61-ac75-7b758308cd13" (UID: "e8286c74-9fa8-4b61-ac75-7b758308cd13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:06.839734 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.839657    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8286c74-9fa8-4b61-ac75-7b758308cd13-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:06.839734 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.839681    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:06.839734 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.839699    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:06.839734 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.839712    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:06.841295 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.840935    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-credential-socket" pod "e8286c74-9fa8-4b61-ac75-7b758308cd13" (UID: "e8286c74-9fa8-4b61-ac75-7b758308cd13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:06.841648 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.841549    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-podinfo" pod "e8286c74-9fa8-4b61-ac75-7b758308cd13" (UID: "e8286c74-9fa8-4b61-ac75-7b758308cd13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:31:06.841752 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.841712    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8286c74-9fa8-4b61-ac75-7b758308cd13-kube-api-access-m29z4" pod "e8286c74-9fa8-4b61-ac75-7b758308cd13" (UID: "e8286c74-9fa8-4b61-ac75-7b758308cd13"). InnerVolumeSpecName "kube-api-access-m29z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:06.843350 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.843317    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-envoy" pod "e8286c74-9fa8-4b61-ac75-7b758308cd13" (UID: "e8286c74-9fa8-4b61-ac75-7b758308cd13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:06.843654 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.843625    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-token" pod "e8286c74-9fa8-4b61-ac75-7b758308cd13" (UID: "e8286c74-9fa8-4b61-ac75-7b758308cd13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:06.942338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.942226    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m29z4\" (UniqueName: \"kubernetes.io/projected/e8286c74-9fa8-4b61-ac75-7b758308cd13-kube-api-access-m29z4\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:06.942338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.942267    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:06.942338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.942282    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:06.942338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.942304    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:06.942338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:06.942335    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8286c74-9fa8-4b61-ac75-7b758308cd13-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:07.694539 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:07.694453    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"
Nov 03 20:31:07.694539 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:07.694483    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm" event={"ID":"e8286c74-9fa8-4b61-ac75-7b758308cd13","Type":"ContainerDied","Data":"d98a4ac19827e169719a419ce34fa2381e584cb506e6dc4c2d14a16c608311a6"}
Nov 03 20:31:07.694999 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:07.694581    5591 scope.go:122] "RemoveContainer" containerID="c6e580505a0c66338d968c496ed003a03c754f43ab1b74f201e79c116b788dff"
Nov 03 20:31:07.766309 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:07.765275    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"]
Nov 03 20:31:07.769995 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:07.769949    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-96230c5b-openshift-default-6bf67c8897-gbdjm"]
Nov 03 20:31:08.658395 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:08.658364    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q"]
Nov 03 20:31:09.352732 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:09.352690    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8286c74-9fa8-4b61-ac75-7b758308cd13" path="/var/lib/kubelet/pods/e8286c74-9fa8-4b61-ac75-7b758308cd13/volumes"
Nov 03 20:31:12.981780 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:12.981743    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"]
Nov 03 20:31:12.982775 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:12.982757    5591 cpu_manager.go:414] "containerMap: removing container" podUID="e8286c74-9fa8-4b61-ac75-7b758308cd13" containerName="istio-proxy"
Nov 03 20:31:12.982841 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:12.982777    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8286c74-9fa8-4b61-ac75-7b758308cd13" containerName="istio-proxy"
Nov 03 20:31:12.982841 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:12.982807    5591 cpu_manager.go:414] "containerMap: removing container" podUID="97472d8f-eb02-4974-9a3d-275a65ca555b" containerName="main"
Nov 03 20:31:12.982841 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:12.982812    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97472d8f-eb02-4974-9a3d-275a65ca555b" containerName="main"
Nov 03 20:31:12.982963 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:12.982953    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="97472d8f-eb02-4974-9a3d-275a65ca555b" containerName="main"
Nov 03 20:31:12.983000 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:12.982976    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8286c74-9fa8-4b61-ac75-7b758308cd13" containerName="istio-proxy"
Nov 03 20:31:12.988616 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:12.988595    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"
Nov 03 20:31:12.996764 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:12.996737    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-836265\"/\"openshift-service-ca.crt\""
Nov 03 20:31:12.997598 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:12.997268    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-836265\"/\"default-dockercfg-ftnn4\""
Nov 03 20:31:12.997779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:12.997386    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-836265\"/\"kube-root-ca.crt\""
Nov 03 20:31:12.997878 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:12.997387    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-836265\"/\"rate-test-model-836265-kserve-self-signed-certs\""
Nov 03 20:31:13.002256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:13.002231    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"]
Nov 03 20:31:13.070798 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:13.070756    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-home\") pod \"rate-test-model-836265-kserve-bd9bbd87c-v4ppw\" (UID: \"143283f7-8182-41d6-bff0-e171c14a2836\") " pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"
Nov 03 20:31:13.070983 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:13.070828    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-model-cache\") pod \"rate-test-model-836265-kserve-bd9bbd87c-v4ppw\" (UID: \"143283f7-8182-41d6-bff0-e171c14a2836\") " pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"
Nov 03 20:31:13.070983 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:13.070939    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w84j\" (UniqueName: \"kubernetes.io/projected/143283f7-8182-41d6-bff0-e171c14a2836-kube-api-access-8w84j\") pod \"rate-test-model-836265-kserve-bd9bbd87c-v4ppw\" (UID: \"143283f7-8182-41d6-bff0-e171c14a2836\") " pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"
Nov 03 20:31:13.071124 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:13.071106    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/143283f7-8182-41d6-bff0-e171c14a2836-tls-certs\") pod \"rate-test-model-836265-kserve-bd9bbd87c-v4ppw\" (UID: \"143283f7-8182-41d6-bff0-e171c14a2836\") " pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"
Nov 03 20:31:13.071190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:13.071176    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-dshm\") pod \"rate-test-model-836265-kserve-bd9bbd87c-v4ppw\" (UID: \"143283f7-8182-41d6-bff0-e171c14a2836\") " pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"
Nov 03 20:31:13.071331 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:13.071313    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-tmp-dir\") pod \"rate-test-model-836265-kserve-bd9bbd87c-v4ppw\" (UID: \"143283f7-8182-41d6-bff0-e171c14a2836\") " pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"
Nov 03 20:31:13.172750 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:13.172704    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-dshm\") pod \"rate-test-model-836265-kserve-bd9bbd87c-v4ppw\" (UID: \"143283f7-8182-41d6-bff0-e171c14a2836\") " pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"
Nov 03 20:31:13.172960 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:13.172823    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-tmp-dir\") pod \"rate-test-model-836265-kserve-bd9bbd87c-v4ppw\" (UID: \"143283f7-8182-41d6-bff0-e171c14a2836\") " pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"
Nov 03 20:31:13.172960 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:13.172904    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-home\") pod \"rate-test-model-836265-kserve-bd9bbd87c-v4ppw\" (UID: \"143283f7-8182-41d6-bff0-e171c14a2836\") " pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"
Nov 03 20:31:13.172960 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:13.172954    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-model-cache\") pod \"rate-test-model-836265-kserve-bd9bbd87c-v4ppw\" (UID: \"143283f7-8182-41d6-bff0-e171c14a2836\") " pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"
Nov 03 20:31:13.173160 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:13.173003    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w84j\" (UniqueName: \"kubernetes.io/projected/143283f7-8182-41d6-bff0-e171c14a2836-kube-api-access-8w84j\") pod \"rate-test-model-836265-kserve-bd9bbd87c-v4ppw\" (UID: \"143283f7-8182-41d6-bff0-e171c14a2836\") " pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"
Nov 03 20:31:13.173160 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:13.173153    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/143283f7-8182-41d6-bff0-e171c14a2836-tls-certs\") pod \"rate-test-model-836265-kserve-bd9bbd87c-v4ppw\" (UID: \"143283f7-8182-41d6-bff0-e171c14a2836\") " pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"
Nov 03 20:31:13.173335 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:13.173284    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-tmp-dir\") pod \"rate-test-model-836265-kserve-bd9bbd87c-v4ppw\" (UID: \"143283f7-8182-41d6-bff0-e171c14a2836\") " pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"
Nov 03 20:31:13.173335 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:13.173311    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-home\") pod \"rate-test-model-836265-kserve-bd9bbd87c-v4ppw\" (UID: \"143283f7-8182-41d6-bff0-e171c14a2836\") " pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"
Nov 03 20:31:13.173478 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:13.173365    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-model-cache\") pod \"rate-test-model-836265-kserve-bd9bbd87c-v4ppw\" (UID: \"143283f7-8182-41d6-bff0-e171c14a2836\") " pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"
Nov 03 20:31:13.175600 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:13.175578    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-dshm\") pod \"rate-test-model-836265-kserve-bd9bbd87c-v4ppw\" (UID: \"143283f7-8182-41d6-bff0-e171c14a2836\") " pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"
Nov 03 20:31:13.175869 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:13.175847    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/143283f7-8182-41d6-bff0-e171c14a2836-tls-certs\") pod \"rate-test-model-836265-kserve-bd9bbd87c-v4ppw\" (UID: \"143283f7-8182-41d6-bff0-e171c14a2836\") " pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"
Nov 03 20:31:13.186383 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:13.186344    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w84j\" (UniqueName: \"kubernetes.io/projected/143283f7-8182-41d6-bff0-e171c14a2836-kube-api-access-8w84j\") pod \"rate-test-model-836265-kserve-bd9bbd87c-v4ppw\" (UID: \"143283f7-8182-41d6-bff0-e171c14a2836\") " pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"
Nov 03 20:31:13.317257 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:13.306394    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"
Nov 03 20:31:13.659648 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:13.659554    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q" podUID="ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd" containerName="payload-pre-processing" containerID="cri-o://7b7caf3dc4d00c8e109a035729a4b41d04308b9c25d078047390001bc8d5add6" gracePeriod=25
Nov 03 20:31:13.689843 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:13.689816    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"]
Nov 03 20:31:13.690013 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:31:13.689983    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod143283f7_8182_41d6_bff0_e171c14a2836.slice/crio-d14ca69abbeb21d7040fe74df4639052af4775d3162e344f722dd065c81e3924 WatchSource:0}: Error finding container d14ca69abbeb21d7040fe74df4639052af4775d3162e344f722dd065c81e3924: Status 404 returned error can't find the container with id d14ca69abbeb21d7040fe74df4639052af4775d3162e344f722dd065c81e3924
Nov 03 20:31:13.756162 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:13.756130    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw" event={"ID":"143283f7-8182-41d6-bff0-e171c14a2836","Type":"ContainerStarted","Data":"d14ca69abbeb21d7040fe74df4639052af4775d3162e344f722dd065c81e3924"}
Nov 03 20:31:14.767198 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:14.767160    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw" event={"ID":"143283f7-8182-41d6-bff0-e171c14a2836","Type":"ContainerStarted","Data":"5ccb476c2ffca7b86d8124a7ea2f2dcc27eef205272b9291e3bf9357c65c3c34"}
Nov 03 20:31:14.789744 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:14.789685    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw" podStartSLOduration=2.789669967 podStartE2EDuration="2.789669967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:14.786910359 +0000 UTC m=+3223.559203368" watchObservedRunningTime="2026-11-03 20:31:14.789669967 +0000 UTC m=+3223.561962984"
Nov 03 20:31:15.307855 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:15.307813    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"
Nov 03 20:31:17.946906 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:17.946861    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87" podUID="666c0175-cb77-4af6-8793-81a7a684c1f2" containerName="main" containerID="cri-o://bab068229c1d465d063b7c4099715f510fbf984434b03a23faaacdd2a5634e0b" gracePeriod=45
Nov 03 20:31:18.282818 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.282789    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"
Nov 03 20:31:18.309615 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.309583    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-dshm\") pod \"666c0175-cb77-4af6-8793-81a7a684c1f2\" (UID: \"666c0175-cb77-4af6-8793-81a7a684c1f2\") "
Nov 03 20:31:18.309615 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.309633    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-home\" (UniqueName: \"kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-home\") pod \"666c0175-cb77-4af6-8793-81a7a684c1f2\" (UID: \"666c0175-cb77-4af6-8793-81a7a684c1f2\") "
Nov 03 20:31:18.310006 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.309751    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/666c0175-cb77-4af6-8793-81a7a684c1f2-kube-api-access-mmrld\" (UniqueName: \"kubernetes.io/projected/666c0175-cb77-4af6-8793-81a7a684c1f2-kube-api-access-mmrld\") pod \"666c0175-cb77-4af6-8793-81a7a684c1f2\" (UID: \"666c0175-cb77-4af6-8793-81a7a684c1f2\") "
Nov 03 20:31:18.310006 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.309865    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-model-cache\") pod \"666c0175-cb77-4af6-8793-81a7a684c1f2\" (UID: \"666c0175-cb77-4af6-8793-81a7a684c1f2\") "
Nov 03 20:31:18.310006 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.309921    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/666c0175-cb77-4af6-8793-81a7a684c1f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/666c0175-cb77-4af6-8793-81a7a684c1f2-tls-certs\") pod \"666c0175-cb77-4af6-8793-81a7a684c1f2\" (UID: \"666c0175-cb77-4af6-8793-81a7a684c1f2\") "
Nov 03 20:31:18.310175 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.310158    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-tmp-dir\") pod \"666c0175-cb77-4af6-8793-81a7a684c1f2\" (UID: \"666c0175-cb77-4af6-8793-81a7a684c1f2\") "
Nov 03 20:31:18.310267 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.310172    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-home" pod "666c0175-cb77-4af6-8793-81a7a684c1f2" (UID: "666c0175-cb77-4af6-8793-81a7a684c1f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:18.310378 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.310348    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-model-cache" pod "666c0175-cb77-4af6-8793-81a7a684c1f2" (UID: "666c0175-cb77-4af6-8793-81a7a684c1f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:18.310677 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.310533    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-tmp-dir" pod "666c0175-cb77-4af6-8793-81a7a684c1f2" (UID: "666c0175-cb77-4af6-8793-81a7a684c1f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:18.311424 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.311393    5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-home\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:18.311542 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.311428    5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-model-cache\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:18.311542 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.311445    5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-tmp-dir\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:18.312478 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.312374    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-dshm" pod "666c0175-cb77-4af6-8793-81a7a684c1f2" (UID: "666c0175-cb77-4af6-8793-81a7a684c1f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:31:18.312574 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.312510    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/666c0175-cb77-4af6-8793-81a7a684c1f2-tls-certs" pod "666c0175-cb77-4af6-8793-81a7a684c1f2" (UID: "666c0175-cb77-4af6-8793-81a7a684c1f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:31:18.312638 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.312583    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/666c0175-cb77-4af6-8793-81a7a684c1f2-kube-api-access-mmrld" pod "666c0175-cb77-4af6-8793-81a7a684c1f2" (UID: "666c0175-cb77-4af6-8793-81a7a684c1f2"). InnerVolumeSpecName "kube-api-access-mmrld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:18.414110 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.414080    5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/666c0175-cb77-4af6-8793-81a7a684c1f2-dshm\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:18.414110 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.414106    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmrld\" (UniqueName: \"kubernetes.io/projected/666c0175-cb77-4af6-8793-81a7a684c1f2-kube-api-access-mmrld\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:18.414110 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.414115    5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/666c0175-cb77-4af6-8793-81a7a684c1f2-tls-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:18.809426 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.809384    5591 generic.go:356] "Generic (PLEG): container finished" podID="666c0175-cb77-4af6-8793-81a7a684c1f2" containerID="bab068229c1d465d063b7c4099715f510fbf984434b03a23faaacdd2a5634e0b" exitCode=0
Nov 03 20:31:18.809426 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.809428    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"
Nov 03 20:31:18.809707 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.809473    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87" event={"ID":"666c0175-cb77-4af6-8793-81a7a684c1f2","Type":"ContainerDied","Data":"bab068229c1d465d063b7c4099715f510fbf984434b03a23faaacdd2a5634e0b"}
Nov 03 20:31:18.809707 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.809518    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87" event={"ID":"666c0175-cb77-4af6-8793-81a7a684c1f2","Type":"ContainerDied","Data":"bf145c0ce1af8f48c7d94eb7255ee6e4dd2052d42b9a2bfb2edc6e47a143aa12"}
Nov 03 20:31:18.809707 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.809541    5591 scope.go:122] "RemoveContainer" containerID="bab068229c1d465d063b7c4099715f510fbf984434b03a23faaacdd2a5634e0b"
Nov 03 20:31:18.825300 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.825283    5591 scope.go:122] "RemoveContainer" containerID="bab068229c1d465d063b7c4099715f510fbf984434b03a23faaacdd2a5634e0b"
Nov 03 20:31:18.825732 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:31:18.825700    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bab068229c1d465d063b7c4099715f510fbf984434b03a23faaacdd2a5634e0b\": container with ID starting with bab068229c1d465d063b7c4099715f510fbf984434b03a23faaacdd2a5634e0b not found: ID does not exist" containerID="bab068229c1d465d063b7c4099715f510fbf984434b03a23faaacdd2a5634e0b"
Nov 03 20:31:18.825832 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.825742    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bab068229c1d465d063b7c4099715f510fbf984434b03a23faaacdd2a5634e0b"} err="failed to get container status \"bab068229c1d465d063b7c4099715f510fbf984434b03a23faaacdd2a5634e0b\": rpc error: code = NotFound desc = could not find container \"bab068229c1d465d063b7c4099715f510fbf984434b03a23faaacdd2a5634e0b\": container with ID starting with bab068229c1d465d063b7c4099715f510fbf984434b03a23faaacdd2a5634e0b not found: ID does not exist"
Nov 03 20:31:18.839398 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.839367    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"]
Nov 03 20:31:18.845029 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:18.844996    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-74dfc6/test-model-74dfc6-kserve-85d9d78f47-zjm87"]
Nov 03 20:31:19.352407 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:19.352371    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="666c0175-cb77-4af6-8793-81a7a684c1f2" path="/var/lib/kubelet/pods/666c0175-cb77-4af6-8793-81a7a684c1f2/volumes"
Nov 03 20:31:22.585915 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:22.585865    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q" podUID="ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:31:23.316232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:23.315695    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"
Nov 03 20:31:24.320233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:24.319303    5591 prober.go:122] "Probe failed" probeType="Startup" pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw" podUID="143283f7-8182-41d6-bff0-e171c14a2836" containerName="main" probeResult="failure" output="Get \"https://10.134.1.3:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:31:24.551277 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:24.550579    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:31:28.727910 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:31:28.727868    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:31:32.585651 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:32.585609    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q" podUID="ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:31:33.319451 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:33.319411    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"
Nov 03 20:31:33.331177 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:33.331147    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"
Nov 03 20:31:33.970307 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:33.970279    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q"
Nov 03 20:31:34.093721 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:34.093688    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd-plugins-config-volume\") pod \"ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd\" (UID: \"ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd\") "
Nov 03 20:31:34.094107 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:34.094081    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd-plugins-config-volume" pod "ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd" (UID: "ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:31:34.094254 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:34.094085    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd-kube-api-access-shgw8\" (UniqueName: \"kubernetes.io/projected/ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd-kube-api-access-shgw8\") pod \"ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd\" (UID: \"ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd\") "
Nov 03 20:31:34.094819 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:34.094798    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:34.096432 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:34.096406    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd-kube-api-access-shgw8" pod "ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd" (UID: "ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd"). InnerVolumeSpecName "kube-api-access-shgw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:31:34.107639 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:34.107603    5591 generic.go:356] "Generic (PLEG): container finished" podID="ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd" containerID="7b7caf3dc4d00c8e109a035729a4b41d04308b9c25d078047390001bc8d5add6" exitCode=0
Nov 03 20:31:34.107810 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:34.107650    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q"
Nov 03 20:31:34.107810 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:34.107677    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q" event={"ID":"ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd","Type":"ContainerDied","Data":"7b7caf3dc4d00c8e109a035729a4b41d04308b9c25d078047390001bc8d5add6"}
Nov 03 20:31:34.107810 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:34.107734    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q" event={"ID":"ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd","Type":"ContainerDied","Data":"8a863a1da93806a121afbcab033d9774d39b3c8a9f42b05994f13374b235164b"}
Nov 03 20:31:34.107810 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:34.107757    5591 scope.go:122] "RemoveContainer" containerID="7b7caf3dc4d00c8e109a035729a4b41d04308b9c25d078047390001bc8d5add6"
Nov 03 20:31:34.134504 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:34.134477    5591 scope.go:122] "RemoveContainer" containerID="7b7caf3dc4d00c8e109a035729a4b41d04308b9c25d078047390001bc8d5add6"
Nov 03 20:31:34.134865 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:31:34.134840    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b7caf3dc4d00c8e109a035729a4b41d04308b9c25d078047390001bc8d5add6\": container with ID starting with 7b7caf3dc4d00c8e109a035729a4b41d04308b9c25d078047390001bc8d5add6 not found: ID does not exist" containerID="7b7caf3dc4d00c8e109a035729a4b41d04308b9c25d078047390001bc8d5add6"
Nov 03 20:31:34.134918 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:34.134874    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b7caf3dc4d00c8e109a035729a4b41d04308b9c25d078047390001bc8d5add6"} err="failed to get container status \"7b7caf3dc4d00c8e109a035729a4b41d04308b9c25d078047390001bc8d5add6\": rpc error: code = NotFound desc = could not find container \"7b7caf3dc4d00c8e109a035729a4b41d04308b9c25d078047390001bc8d5add6\": container with ID starting with 7b7caf3dc4d00c8e109a035729a4b41d04308b9c25d078047390001bc8d5add6 not found: ID does not exist"
Nov 03 20:31:34.155165 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:34.155128    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q"]
Nov 03 20:31:34.159389 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:34.159368    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7b70d-746cc7f89f-gjp7q"]
Nov 03 20:31:34.196928 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:34.196841    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shgw8\" (UniqueName: \"kubernetes.io/projected/ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd-kube-api-access-shgw8\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:31:35.354156 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:35.354123    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd" path="/var/lib/kubelet/pods/ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd/volumes"
Nov 03 20:31:37.938516 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:37.938482    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"]
Nov 03 20:31:37.940035 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:37.940009    5591 cpu_manager.go:414] "containerMap: removing container" podUID="ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd" containerName="payload-pre-processing"
Nov 03 20:31:37.940035 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:37.940035    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd" containerName="payload-pre-processing"
Nov 03 20:31:37.940249 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:37.940137    5591 cpu_manager.go:414] "containerMap: removing container" podUID="666c0175-cb77-4af6-8793-81a7a684c1f2" containerName="main"
Nov 03 20:31:37.940249 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:37.940146    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="666c0175-cb77-4af6-8793-81a7a684c1f2" containerName="main"
Nov 03 20:31:37.940415 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:37.940400    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="666c0175-cb77-4af6-8793-81a7a684c1f2" containerName="main"
Nov 03 20:31:37.940489 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:37.940428    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed8d7bd5-8fb7-4f6e-b986-9fa1480d8bfd" containerName="payload-pre-processing"
Nov 03 20:31:37.950494 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:37.950464    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"
Nov 03 20:31:37.955276 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:37.955227    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"]
Nov 03 20:31:37.958972 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:37.958944    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-660d95\"/\"kube-root-ca.crt\""
Nov 03 20:31:37.959157 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:37.959018    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-660d95\"/\"openshift-service-ca.crt\""
Nov 03 20:31:37.959157 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:37.958946    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-660d95\"/\"default-dockercfg-6w4sj\""
Nov 03 20:31:37.959157 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:37.959052    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-660d95\"/\"rate-test-model-660d95-kserve-self-signed-certs\""
Nov 03 20:31:38.004404 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:38.004368    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-home\") pod \"rate-test-model-660d95-kserve-65df4f7c57-xj8jh\" (UID: \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\") " pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"
Nov 03 20:31:38.004821 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:38.004791    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-dshm\") pod \"rate-test-model-660d95-kserve-65df4f7c57-xj8jh\" (UID: \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\") " pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"
Nov 03 20:31:38.004938 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:38.004843    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-model-cache\") pod \"rate-test-model-660d95-kserve-65df4f7c57-xj8jh\" (UID: \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\") " pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"
Nov 03 20:31:38.005176 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:38.005159    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-tmp-dir\") pod \"rate-test-model-660d95-kserve-65df4f7c57-xj8jh\" (UID: \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\") " pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"
Nov 03 20:31:38.005393 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:38.005365    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-tls-certs\") pod \"rate-test-model-660d95-kserve-65df4f7c57-xj8jh\" (UID: \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\") " pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"
Nov 03 20:31:38.005500 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:38.005408    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4blwh\" (UniqueName: \"kubernetes.io/projected/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-kube-api-access-4blwh\") pod \"rate-test-model-660d95-kserve-65df4f7c57-xj8jh\" (UID: \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\") " pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"
Nov 03 20:31:38.131231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:38.130712    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-dshm\") pod \"rate-test-model-660d95-kserve-65df4f7c57-xj8jh\" (UID: \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\") " pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"
Nov 03 20:31:38.131231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:38.130767    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-model-cache\") pod \"rate-test-model-660d95-kserve-65df4f7c57-xj8jh\" (UID: \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\") " pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"
Nov 03 20:31:38.131231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:38.131113    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-tmp-dir\") pod \"rate-test-model-660d95-kserve-65df4f7c57-xj8jh\" (UID: \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\") " pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"
Nov 03 20:31:38.131567 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:38.131272    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-tls-certs\") pod \"rate-test-model-660d95-kserve-65df4f7c57-xj8jh\" (UID: \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\") " pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"
Nov 03 20:31:38.131567 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:38.131299    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4blwh\" (UniqueName: \"kubernetes.io/projected/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-kube-api-access-4blwh\") pod \"rate-test-model-660d95-kserve-65df4f7c57-xj8jh\" (UID: \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\") " pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"
Nov 03 20:31:38.131567 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:38.131385    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-home\") pod \"rate-test-model-660d95-kserve-65df4f7c57-xj8jh\" (UID: \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\") " pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"
Nov 03 20:31:38.148228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:38.131971    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-home\") pod \"rate-test-model-660d95-kserve-65df4f7c57-xj8jh\" (UID: \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\") " pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"
Nov 03 20:31:38.154227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:38.151665    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-tmp-dir\") pod \"rate-test-model-660d95-kserve-65df4f7c57-xj8jh\" (UID: \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\") " pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"
Nov 03 20:31:38.154227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:38.151929    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-model-cache\") pod \"rate-test-model-660d95-kserve-65df4f7c57-xj8jh\" (UID: \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\") " pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"
Nov 03 20:31:38.199086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:38.198965    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-tls-certs\") pod \"rate-test-model-660d95-kserve-65df4f7c57-xj8jh\" (UID: \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\") " pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"
Nov 03 20:31:38.208910 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:38.208872    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-dshm\") pod \"rate-test-model-660d95-kserve-65df4f7c57-xj8jh\" (UID: \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\") " pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"
Nov 03 20:31:38.218413 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:38.209820    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4blwh\" (UniqueName: \"kubernetes.io/projected/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-kube-api-access-4blwh\") pod \"rate-test-model-660d95-kserve-65df4f7c57-xj8jh\" (UID: \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\") " pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"
Nov 03 20:31:38.275706 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:38.275645    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"
Nov 03 20:31:38.615393 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:38.615358    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"]
Nov 03 20:31:38.617174 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:31:38.617141    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode10f31e1_5a95_4c6a_88ed_0d15291cb44e.slice/crio-d62049e23135f2ce8da8ebf9a4da46f4015185e29477da36bd0f5d89da56aa30 WatchSource:0}: Error finding container d62049e23135f2ce8da8ebf9a4da46f4015185e29477da36bd0f5d89da56aa30: Status 404 returned error can't find the container with id d62049e23135f2ce8da8ebf9a4da46f4015185e29477da36bd0f5d89da56aa30
Nov 03 20:31:39.242654 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:39.242611    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh" event={"ID":"e10f31e1-5a95-4c6a-88ed-0d15291cb44e","Type":"ContainerStarted","Data":"d51f6dfd474b025fbd9965f0367235b3acf497eacc0c4b619fb840bb4ff742da"}
Nov 03 20:31:39.242654 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:39.242667    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh" event={"ID":"e10f31e1-5a95-4c6a-88ed-0d15291cb44e","Type":"ContainerStarted","Data":"d62049e23135f2ce8da8ebf9a4da46f4015185e29477da36bd0f5d89da56aa30"}
Nov 03 20:31:39.267867 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:39.267807    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh" podStartSLOduration=2.267792784 podStartE2EDuration="2.267792784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:39.264777899 +0000 UTC m=+3248.037070917" watchObservedRunningTime="2026-11-03 20:31:39.267792784 +0000 UTC m=+3248.040085802"
Nov 03 20:31:39.275833 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:39.275808    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"
Nov 03 20:31:48.276444 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:48.276341    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"
Nov 03 20:31:48.292264 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:48.292196    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"
Nov 03 20:31:48.335147 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:31:48.335121    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"
Nov 03 20:31:58.757113 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:31:58.757069    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:32:02.446767 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.446730    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr"]
Nov 03 20:32:02.453380 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.453351    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr"
Nov 03 20:32:02.472829 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.472802    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d5cf239f-c014-4bfe-9479-88a51dbccff3-data\") pod \"postgres-7cdbf6dc44-4jqgr\" (UID: \"d5cf239f-c014-4bfe-9479-88a51dbccff3\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr"
Nov 03 20:32:02.473055 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.473033    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzl9q\" (UniqueName: \"kubernetes.io/projected/d5cf239f-c014-4bfe-9479-88a51dbccff3-kube-api-access-rzl9q\") pod \"postgres-7cdbf6dc44-4jqgr\" (UID: \"d5cf239f-c014-4bfe-9479-88a51dbccff3\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr"
Nov 03 20:32:02.481045 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.481013    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 03 20:32:02.481295 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.481275    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-w84zh\""
Nov 03 20:32:02.512823 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.512790    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb"]
Nov 03 20:32:02.529305 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.529275    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr"]
Nov 03 20:32:02.529571 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.529332    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb"
Nov 03 20:32:02.547355 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.547326    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-32e482\""
Nov 03 20:32:02.574600 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.574570    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-852jw\" (UniqueName: \"kubernetes.io/projected/a97c5397-3fc1-46f5-9627-9681bcab980b-kube-api-access-852jw\") pod \"payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb\" (UID: \"a97c5397-3fc1-46f5-9627-9681bcab980b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb"
Nov 03 20:32:02.574806 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.574782    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzl9q\" (UniqueName: \"kubernetes.io/projected/d5cf239f-c014-4bfe-9479-88a51dbccff3-kube-api-access-rzl9q\") pod \"postgres-7cdbf6dc44-4jqgr\" (UID: \"d5cf239f-c014-4bfe-9479-88a51dbccff3\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr"
Nov 03 20:32:02.575016 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.574993    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a97c5397-3fc1-46f5-9627-9681bcab980b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb\" (UID: \"a97c5397-3fc1-46f5-9627-9681bcab980b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb"
Nov 03 20:32:02.575115 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.575098    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d5cf239f-c014-4bfe-9479-88a51dbccff3-data\") pod \"postgres-7cdbf6dc44-4jqgr\" (UID: \"d5cf239f-c014-4bfe-9479-88a51dbccff3\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr"
Nov 03 20:32:02.575657 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.575632    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d5cf239f-c014-4bfe-9479-88a51dbccff3-data\") pod \"postgres-7cdbf6dc44-4jqgr\" (UID: \"d5cf239f-c014-4bfe-9479-88a51dbccff3\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr"
Nov 03 20:32:02.578550 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.578528    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-32e482-dockercfg-4mbnz\""
Nov 03 20:32:02.594863 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.594837    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzl9q\" (UniqueName: \"kubernetes.io/projected/d5cf239f-c014-4bfe-9479-88a51dbccff3-kube-api-access-rzl9q\") pod \"postgres-7cdbf6dc44-4jqgr\" (UID: \"d5cf239f-c014-4bfe-9479-88a51dbccff3\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr"
Nov 03 20:32:02.626781 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.626745    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6x2hv"]
Nov 03 20:32:02.635703 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.635658    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb"]
Nov 03 20:32:02.635902 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.635812    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6x2hv"
Nov 03 20:32:02.674371 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.674343    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 03 20:32:02.674563 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.674468    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-h4l58\""
Nov 03 20:32:02.677072 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.676759    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a97c5397-3fc1-46f5-9627-9681bcab980b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb\" (UID: \"a97c5397-3fc1-46f5-9627-9681bcab980b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb"
Nov 03 20:32:02.677072 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.676914    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/dc03a37b-a84b-4df5-9306-57b8e1ec8f7b-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-6x2hv\" (UID: \"dc03a37b-a84b-4df5-9306-57b8e1ec8f7b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6x2hv"
Nov 03 20:32:02.677072 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.677045    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-852jw\" (UniqueName: \"kubernetes.io/projected/a97c5397-3fc1-46f5-9627-9681bcab980b-kube-api-access-852jw\") pod \"payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb\" (UID: \"a97c5397-3fc1-46f5-9627-9681bcab980b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb"
Nov 03 20:32:02.677398 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.677375    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6x2hv"]
Nov 03 20:32:02.677814 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.677775    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a97c5397-3fc1-46f5-9627-9681bcab980b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb\" (UID: \"a97c5397-3fc1-46f5-9627-9681bcab980b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb"
Nov 03 20:32:02.682559 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.682477    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk"]
Nov 03 20:32:02.683187 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.683093    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk" podUID="07a28774-81b4-4051-9e4f-cb8fbe703942" containerName="maas-api" containerID="cri-o://59a40d58c3df89999f733881b03c925eb23f1bcbeecc41e8aa0cf9bdb6d3103b" gracePeriod=30
Nov 03 20:32:02.690842 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.690351    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"]
Nov 03 20:32:02.690842 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.690605    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s" podUID="f51a1214-6af0-4ce3-b924-ca8d06c860fc" containerName="istio-proxy" containerID="cri-o://93fa05cccc54e0d6a2772a1eb9120569c678c347d5d5600b78af40654c5b06b9" gracePeriod=30
Nov 03 20:32:02.693648 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.693624    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"]
Nov 03 20:32:02.693836 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.693812    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn" podUID="ea34ecc5-6c75-44b0-a28a-530668e79e47" containerName="istio-proxy" containerID="cri-o://9bc089cd5e57f1afac4279e59f62b6e8f7bc7cc4671af1bc3079c826e3dcb246" gracePeriod=30
Nov 03 20:32:02.700121 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.699601    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"]
Nov 03 20:32:02.700121 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.700024    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" containerID="cri-o://c682b4964142f5e0b9b5103c615b3529905b71dd682fccfc7e7593ab45717cf3" gracePeriod=30
Nov 03 20:32:02.726532 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.725789    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-852jw\" (UniqueName: \"kubernetes.io/projected/a97c5397-3fc1-46f5-9627-9681bcab980b-kube-api-access-852jw\") pod \"payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb\" (UID: \"a97c5397-3fc1-46f5-9627-9681bcab980b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb"
Nov 03 20:32:02.729534 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.729505    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"]
Nov 03 20:32:02.742276 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.736458    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr"]
Nov 03 20:32:02.742276 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.736584    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"
Nov 03 20:32:02.742276 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.737172    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr"
Nov 03 20:32:02.758961 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.758882    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"]
Nov 03 20:32:02.782372 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.782334    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3fbb1a\"/\"simulator-w4-3fbb1a-kserve-self-signed-certs\""
Nov 03 20:32:02.786431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.785568    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4bgt\" (UniqueName: \"kubernetes.io/projected/ea3c9bb0-fa05-4815-8412-6db003469cd0-kube-api-access-m4bgt\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-5rp46\" (UID: \"ea3c9bb0-fa05-4815-8412-6db003469cd0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"
Nov 03 20:32:02.786431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.785634    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea3c9bb0-fa05-4815-8412-6db003469cd0-tls-certs\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-5rp46\" (UID: \"ea3c9bb0-fa05-4815-8412-6db003469cd0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"
Nov 03 20:32:02.786431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.785762    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-dshm\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-5rp46\" (UID: \"ea3c9bb0-fa05-4815-8412-6db003469cd0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"
Nov 03 20:32:02.786431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.785788    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-model-cache\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-5rp46\" (UID: \"ea3c9bb0-fa05-4815-8412-6db003469cd0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"
Nov 03 20:32:02.786431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.785848    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-home\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-5rp46\" (UID: \"ea3c9bb0-fa05-4815-8412-6db003469cd0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"
Nov 03 20:32:02.786431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.785889    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/dc03a37b-a84b-4df5-9306-57b8e1ec8f7b-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-6x2hv\" (UID: \"dc03a37b-a84b-4df5-9306-57b8e1ec8f7b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6x2hv"
Nov 03 20:32:02.786431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.786309    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-tmp-dir\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-5rp46\" (UID: \"ea3c9bb0-fa05-4815-8412-6db003469cd0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"
Nov 03 20:32:02.794662 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.794630    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/dc03a37b-a84b-4df5-9306-57b8e1ec8f7b-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-6x2hv\" (UID: \"dc03a37b-a84b-4df5-9306-57b8e1ec8f7b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6x2hv"
Nov 03 20:32:02.880532 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.880279    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb"
Nov 03 20:32:02.903508 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.897651    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-tmp-dir\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-5rp46\" (UID: \"ea3c9bb0-fa05-4815-8412-6db003469cd0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"
Nov 03 20:32:02.903508 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.897737    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4bgt\" (UniqueName: \"kubernetes.io/projected/ea3c9bb0-fa05-4815-8412-6db003469cd0-kube-api-access-m4bgt\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-5rp46\" (UID: \"ea3c9bb0-fa05-4815-8412-6db003469cd0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"
Nov 03 20:32:02.903508 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.897791    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea3c9bb0-fa05-4815-8412-6db003469cd0-tls-certs\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-5rp46\" (UID: \"ea3c9bb0-fa05-4815-8412-6db003469cd0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"
Nov 03 20:32:02.903508 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.903436    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-tmp-dir\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-5rp46\" (UID: \"ea3c9bb0-fa05-4815-8412-6db003469cd0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"
Nov 03 20:32:02.961149 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.960847    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-dshm\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-5rp46\" (UID: \"ea3c9bb0-fa05-4815-8412-6db003469cd0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"
Nov 03 20:32:02.961801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.961653    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-model-cache\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-5rp46\" (UID: \"ea3c9bb0-fa05-4815-8412-6db003469cd0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"
Nov 03 20:32:02.962057 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.962041    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-home\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-5rp46\" (UID: \"ea3c9bb0-fa05-4815-8412-6db003469cd0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"
Nov 03 20:32:02.963485 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.963464    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-home\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-5rp46\" (UID: \"ea3c9bb0-fa05-4815-8412-6db003469cd0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"
Nov 03 20:32:02.964892 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:02.964594    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-model-cache\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-5rp46\" (UID: \"ea3c9bb0-fa05-4815-8412-6db003469cd0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"
Nov 03 20:32:03.088149 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.085883    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea3c9bb0-fa05-4815-8412-6db003469cd0-tls-certs\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-5rp46\" (UID: \"ea3c9bb0-fa05-4815-8412-6db003469cd0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"
Nov 03 20:32:03.088149 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.086173    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-dshm\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-5rp46\" (UID: \"ea3c9bb0-fa05-4815-8412-6db003469cd0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"
Nov 03 20:32:03.088149 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.086280    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6x2hv"
Nov 03 20:32:03.088149 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.086806    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4bgt\" (UniqueName: \"kubernetes.io/projected/ea3c9bb0-fa05-4815-8412-6db003469cd0-kube-api-access-m4bgt\") pod \"simulator-w4-3fbb1a-kserve-7868585bbb-5rp46\" (UID: \"ea3c9bb0-fa05-4815-8412-6db003469cd0\") " pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"
Nov 03 20:32:03.146809 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.146762    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"]
Nov 03 20:32:03.157767 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.157735    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"
Nov 03 20:32:03.171901 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.171867    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9f46e5\"/\"simulator-w3-9f46e5-kserve-self-signed-certs\""
Nov 03 20:32:03.173187 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.173157    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"]
Nov 03 20:32:03.223026 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.222703    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-model-cache\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh\" (UID: \"665623ee-f9a4-4aae-922b-f229460a9cf2\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"
Nov 03 20:32:03.223601 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.223562    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-home\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh\" (UID: \"665623ee-f9a4-4aae-922b-f229460a9cf2\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"
Nov 03 20:32:03.223736 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.223699    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-tmp-dir\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh\" (UID: \"665623ee-f9a4-4aae-922b-f229460a9cf2\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"
Nov 03 20:32:03.223799 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.223764    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/665623ee-f9a4-4aae-922b-f229460a9cf2-tls-certs\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh\" (UID: \"665623ee-f9a4-4aae-922b-f229460a9cf2\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"
Nov 03 20:32:03.223799 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.223790    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-dshm\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh\" (UID: \"665623ee-f9a4-4aae-922b-f229460a9cf2\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"
Nov 03 20:32:03.223868 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.223832    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqw6n\" (UniqueName: \"kubernetes.io/projected/665623ee-f9a4-4aae-922b-f229460a9cf2-kube-api-access-vqw6n\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh\" (UID: \"665623ee-f9a4-4aae-922b-f229460a9cf2\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"
Nov 03 20:32:03.260887 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.260837    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"
Nov 03 20:32:03.326877 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.326837    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-home\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh\" (UID: \"665623ee-f9a4-4aae-922b-f229460a9cf2\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"
Nov 03 20:32:03.327059 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.326922    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-tmp-dir\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh\" (UID: \"665623ee-f9a4-4aae-922b-f229460a9cf2\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"
Nov 03 20:32:03.327059 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.326983    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/665623ee-f9a4-4aae-922b-f229460a9cf2-tls-certs\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh\" (UID: \"665623ee-f9a4-4aae-922b-f229460a9cf2\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"
Nov 03 20:32:03.327187 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.327110    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-dshm\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh\" (UID: \"665623ee-f9a4-4aae-922b-f229460a9cf2\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"
Nov 03 20:32:03.327362 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.327336    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-home\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh\" (UID: \"665623ee-f9a4-4aae-922b-f229460a9cf2\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"
Nov 03 20:32:03.327427 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.327382    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqw6n\" (UniqueName: \"kubernetes.io/projected/665623ee-f9a4-4aae-922b-f229460a9cf2-kube-api-access-vqw6n\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh\" (UID: \"665623ee-f9a4-4aae-922b-f229460a9cf2\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"
Nov 03 20:32:03.327701 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.327679    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-model-cache\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh\" (UID: \"665623ee-f9a4-4aae-922b-f229460a9cf2\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"
Nov 03 20:32:03.328288 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.328243    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-model-cache\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh\" (UID: \"665623ee-f9a4-4aae-922b-f229460a9cf2\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"
Nov 03 20:32:03.328394 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.328341    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-tmp-dir\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh\" (UID: \"665623ee-f9a4-4aae-922b-f229460a9cf2\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"
Nov 03 20:32:03.341419 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.338858    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-dshm\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh\" (UID: \"665623ee-f9a4-4aae-922b-f229460a9cf2\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"
Nov 03 20:32:03.341419 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.339629    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/665623ee-f9a4-4aae-922b-f229460a9cf2-tls-certs\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh\" (UID: \"665623ee-f9a4-4aae-922b-f229460a9cf2\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"
Nov 03 20:32:03.341419 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.340870    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr"]
Nov 03 20:32:03.346075 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.346016    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqw6n\" (UniqueName: \"kubernetes.io/projected/665623ee-f9a4-4aae-922b-f229460a9cf2-kube-api-access-vqw6n\") pod \"simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh\" (UID: \"665623ee-f9a4-4aae-922b-f229460a9cf2\") " pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"
Nov 03 20:32:03.349986 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.349456    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk"
Nov 03 20:32:03.428905 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.428875    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07a28774-81b4-4051-9e4f-cb8fbe703942-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07a28774-81b4-4051-9e4f-cb8fbe703942-maas-api-tls\") pod \"07a28774-81b4-4051-9e4f-cb8fbe703942\" (UID: \"07a28774-81b4-4051-9e4f-cb8fbe703942\") "
Nov 03 20:32:03.429111 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.428917    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a28774-81b4-4051-9e4f-cb8fbe703942-kube-api-access-jfw44\" (UniqueName: \"kubernetes.io/projected/07a28774-81b4-4051-9e4f-cb8fbe703942-kube-api-access-jfw44\") pod \"07a28774-81b4-4051-9e4f-cb8fbe703942\" (UID: \"07a28774-81b4-4051-9e4f-cb8fbe703942\") "
Nov 03 20:32:03.429111 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.428988    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a28774-81b4-4051-9e4f-cb8fbe703942-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a28774-81b4-4051-9e4f-cb8fbe703942-openshift-service-ca\") pod \"07a28774-81b4-4051-9e4f-cb8fbe703942\" (UID: \"07a28774-81b4-4051-9e4f-cb8fbe703942\") "
Nov 03 20:32:03.430793 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.429379    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07a28774-81b4-4051-9e4f-cb8fbe703942-metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a28774-81b4-4051-9e4f-cb8fbe703942-metrics-certs\") pod \"07a28774-81b4-4051-9e4f-cb8fbe703942\" (UID: \"07a28774-81b4-4051-9e4f-cb8fbe703942\") "
Nov 03 20:32:03.430793 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.429443    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a28774-81b4-4051-9e4f-cb8fbe703942-openshift-service-ca" pod "07a28774-81b4-4051-9e4f-cb8fbe703942" (UID: "07a28774-81b4-4051-9e4f-cb8fbe703942"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:03.430938 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.430876    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a28774-81b4-4051-9e4f-cb8fbe703942-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:03.432630 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.432605    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07a28774-81b4-4051-9e4f-cb8fbe703942-metrics-certs" pod "07a28774-81b4-4051-9e4f-cb8fbe703942" (UID: "07a28774-81b4-4051-9e4f-cb8fbe703942"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:32:03.432740 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.432707    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07a28774-81b4-4051-9e4f-cb8fbe703942-maas-api-tls" pod "07a28774-81b4-4051-9e4f-cb8fbe703942" (UID: "07a28774-81b4-4051-9e4f-cb8fbe703942"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:32:03.434912 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.434803    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a28774-81b4-4051-9e4f-cb8fbe703942-kube-api-access-jfw44" pod "07a28774-81b4-4051-9e4f-cb8fbe703942" (UID: "07a28774-81b4-4051-9e4f-cb8fbe703942"). InnerVolumeSpecName "kube-api-access-jfw44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:03.465403 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.465363    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb"]
Nov 03 20:32:03.471574 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.471546    5591 generic.go:356] "Generic (PLEG): container finished" podID="07a28774-81b4-4051-9e4f-cb8fbe703942" containerID="59a40d58c3df89999f733881b03c925eb23f1bcbeecc41e8aa0cf9bdb6d3103b" exitCode=0
Nov 03 20:32:03.471744 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.471593    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk"
Nov 03 20:32:03.471744 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.471633    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk" event={"ID":"07a28774-81b4-4051-9e4f-cb8fbe703942","Type":"ContainerDied","Data":"59a40d58c3df89999f733881b03c925eb23f1bcbeecc41e8aa0cf9bdb6d3103b"}
Nov 03 20:32:03.471744 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.471687    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk" event={"ID":"07a28774-81b4-4051-9e4f-cb8fbe703942","Type":"ContainerDied","Data":"f03a9fd2319a73a70d6f8267bb50b7fa5b292207d7b8c66b53dfff2623f109f9"}
Nov 03 20:32:03.471744 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.471717    5591 scope.go:122] "RemoveContainer" containerID="59a40d58c3df89999f733881b03c925eb23f1bcbeecc41e8aa0cf9bdb6d3103b"
Nov 03 20:32:03.475983 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.475916    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr" event={"ID":"d5cf239f-c014-4bfe-9479-88a51dbccff3","Type":"ContainerStarted","Data":"b7cdd6f944f6dd926826b9401e050374d874e37f7074a2c86b352cb400deb238"}
Nov 03 20:32:03.490355 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.490330    5591 scope.go:122] "RemoveContainer" containerID="59a40d58c3df89999f733881b03c925eb23f1bcbeecc41e8aa0cf9bdb6d3103b"
Nov 03 20:32:03.491007 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:03.490977    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59a40d58c3df89999f733881b03c925eb23f1bcbeecc41e8aa0cf9bdb6d3103b\": container with ID starting with 59a40d58c3df89999f733881b03c925eb23f1bcbeecc41e8aa0cf9bdb6d3103b not found: ID does not exist" containerID="59a40d58c3df89999f733881b03c925eb23f1bcbeecc41e8aa0cf9bdb6d3103b"
Nov 03 20:32:03.491125 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.491016    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59a40d58c3df89999f733881b03c925eb23f1bcbeecc41e8aa0cf9bdb6d3103b"} err="failed to get container status \"59a40d58c3df89999f733881b03c925eb23f1bcbeecc41e8aa0cf9bdb6d3103b\": rpc error: code = NotFound desc = could not find container \"59a40d58c3df89999f733881b03c925eb23f1bcbeecc41e8aa0cf9bdb6d3103b\": container with ID starting with 59a40d58c3df89999f733881b03c925eb23f1bcbeecc41e8aa0cf9bdb6d3103b not found: ID does not exist"
Nov 03 20:32:03.496981 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:32:03.496936    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc03a37b_a84b_4df5_9306_57b8e1ec8f7b.slice/crio-4e7320663f6879ec6108c1246fa65993a1367b46ebc1e7ea297862f084662af5 WatchSource:0}: Error finding container 4e7320663f6879ec6108c1246fa65993a1367b46ebc1e7ea297862f084662af5: Status 404 returned error can't find the container with id 4e7320663f6879ec6108c1246fa65993a1367b46ebc1e7ea297862f084662af5
Nov 03 20:32:03.497083 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.497015    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6x2hv"]
Nov 03 20:32:03.498166 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:32:03.498142    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97c5397_3fc1_46f5_9627_9681bcab980b.slice/crio-5976d8eeceef17fb1d8c88cdb5364793395fc18c4a6c9a99a27176e3fee67590 WatchSource:0}: Error finding container 5976d8eeceef17fb1d8c88cdb5364793395fc18c4a6c9a99a27176e3fee67590: Status 404 returned error can't find the container with id 5976d8eeceef17fb1d8c88cdb5364793395fc18c4a6c9a99a27176e3fee67590
Nov 03 20:32:03.503239 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.503024    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb"]
Nov 03 20:32:03.505534 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.505502    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"
Nov 03 20:32:03.532859 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.532512    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a28774-81b4-4051-9e4f-cb8fbe703942-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:03.532859 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.532546    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07a28774-81b4-4051-9e4f-cb8fbe703942-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:03.532859 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.532563    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfw44\" (UniqueName: \"kubernetes.io/projected/07a28774-81b4-4051-9e4f-cb8fbe703942-kube-api-access-jfw44\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:03.538294 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.537943    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk"]
Nov 03 20:32:03.548502 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.548476    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-74dfc6-5b767f49b4-kg6bk"]
Nov 03 20:32:03.628907 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.628873    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"]
Nov 03 20:32:03.629996 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:32:03.629962    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea3c9bb0_fa05_4815_8412_6db003469cd0.slice/crio-1ed12399ca1568995adebd2f8a5c29efa4f2e627d7819d02d2ce0cc1d820f684 WatchSource:0}: Error finding container 1ed12399ca1568995adebd2f8a5c29efa4f2e627d7819d02d2ce0cc1d820f684: Status 404 returned error can't find the container with id 1ed12399ca1568995adebd2f8a5c29efa4f2e627d7819d02d2ce0cc1d820f684
Nov 03 20:32:03.907854 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:32:03.907818    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod665623ee_f9a4_4aae_922b_f229460a9cf2.slice/crio-9c144af027b1af358d1dd753254f35896800722063b41f840223821306edb1e9 WatchSource:0}: Error finding container 9c144af027b1af358d1dd753254f35896800722063b41f840223821306edb1e9: Status 404 returned error can't find the container with id 9c144af027b1af358d1dd753254f35896800722063b41f840223821306edb1e9
Nov 03 20:32:03.908746 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:03.908707    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"]
Nov 03 20:32:04.154491 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.154456    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"]
Nov 03 20:32:04.154743 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.154701    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg" podUID="53641647-e68c-4a85-8d1a-fa903d2acdad" containerName="istio-proxy" containerID="cri-o://d3f75afa34deb64b495136125a529cffcb613a97568ccb7074588bfb377923ec" gracePeriod=30
Nov 03 20:32:04.377956 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.377912    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf"]
Nov 03 20:32:04.379075 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.379051    5591 cpu_manager.go:414] "containerMap: removing container" podUID="07a28774-81b4-4051-9e4f-cb8fbe703942" containerName="maas-api"
Nov 03 20:32:04.379075 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.379073    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a28774-81b4-4051-9e4f-cb8fbe703942" containerName="maas-api"
Nov 03 20:32:04.379319 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.379289    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a28774-81b4-4051-9e4f-cb8fbe703942" containerName="maas-api"
Nov 03 20:32:04.383953 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.383932    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf"
Nov 03 20:32:04.404138 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.404114    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf"]
Nov 03 20:32:04.445340 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.445301    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"]
Nov 03 20:32:04.451381 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.451352    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"
Nov 03 20:32:04.452028 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.452003    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"]
Nov 03 20:32:04.457703 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.457671    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/956c41e5-c95c-4dc4-8734-99bd8cecd1be-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf\" (UID: \"956c41e5-c95c-4dc4-8734-99bd8cecd1be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf"
Nov 03 20:32:04.457895 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.457761    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/956c41e5-c95c-4dc4-8734-99bd8cecd1be-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf\" (UID: \"956c41e5-c95c-4dc4-8734-99bd8cecd1be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf"
Nov 03 20:32:04.457895 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.457795    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7stdz\" (UniqueName: \"kubernetes.io/projected/956c41e5-c95c-4dc4-8734-99bd8cecd1be-kube-api-access-7stdz\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf\" (UID: \"956c41e5-c95c-4dc4-8734-99bd8cecd1be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf"
Nov 03 20:32:04.458108 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.457969    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/956c41e5-c95c-4dc4-8734-99bd8cecd1be-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf\" (UID: \"956c41e5-c95c-4dc4-8734-99bd8cecd1be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf"
Nov 03 20:32:04.458108 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.458044    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"
Nov 03 20:32:04.458273 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.458143    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/956c41e5-c95c-4dc4-8734-99bd8cecd1be-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf\" (UID: \"956c41e5-c95c-4dc4-8734-99bd8cecd1be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf"
Nov 03 20:32:04.458273 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.458227    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/956c41e5-c95c-4dc4-8734-99bd8cecd1be-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf\" (UID: \"956c41e5-c95c-4dc4-8734-99bd8cecd1be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf"
Nov 03 20:32:04.468062 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.467922    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-32e482\"/\"simulator-w1-32e482-kserve-self-signed-certs\""
Nov 03 20:32:04.469088 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.468570    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 03 20:32:04.469088 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.468702    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2b5703\"/\"distinct-w2-2b5703-kserve-self-signed-certs\""
Nov 03 20:32:04.469088 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.468985    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"]
Nov 03 20:32:04.475788 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.475748    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"]
Nov 03 20:32:04.510731 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.510692    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb" event={"ID":"a97c5397-3fc1-46f5-9627-9681bcab980b","Type":"ContainerStarted","Data":"a4704e73b5cf7d9da8837491c2fb4d79d616f1cb76f076a8f861dbb1a8af690f"}
Nov 03 20:32:04.511329 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.511295    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb" event={"ID":"a97c5397-3fc1-46f5-9627-9681bcab980b","Type":"ContainerStarted","Data":"5976d8eeceef17fb1d8c88cdb5364793395fc18c4a6c9a99a27176e3fee67590"}
Nov 03 20:32:04.511781 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.511756    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb"
Nov 03 20:32:04.514400 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.514377    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46" event={"ID":"ea3c9bb0-fa05-4815-8412-6db003469cd0","Type":"ContainerStarted","Data":"d67af5494e52ed57f05c266adfb74510944c6479d0ee6ccd5adffd1ff21a4513"}
Nov 03 20:32:04.514559 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.514542    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46" event={"ID":"ea3c9bb0-fa05-4815-8412-6db003469cd0","Type":"ContainerStarted","Data":"1ed12399ca1568995adebd2f8a5c29efa4f2e627d7819d02d2ce0cc1d820f684"}
Nov 03 20:32:04.518517 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.518493    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6x2hv" event={"ID":"dc03a37b-a84b-4df5-9306-57b8e1ec8f7b","Type":"ContainerStarted","Data":"4e7320663f6879ec6108c1246fa65993a1367b46ebc1e7ea297862f084662af5"}
Nov 03 20:32:04.522173 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.522151    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh" event={"ID":"665623ee-f9a4-4aae-922b-f229460a9cf2","Type":"ContainerStarted","Data":"036a29cf2b55fe80c230491d6447b63f0f5cabb2fe4ebffa140e25fad7587edf"}
Nov 03 20:32:04.522303 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.522181    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh" event={"ID":"665623ee-f9a4-4aae-922b-f229460a9cf2","Type":"ContainerStarted","Data":"9c144af027b1af358d1dd753254f35896800722063b41f840223821306edb1e9"}
Nov 03 20:32:04.580472 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.580428    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/956c41e5-c95c-4dc4-8734-99bd8cecd1be-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf\" (UID: \"956c41e5-c95c-4dc4-8734-99bd8cecd1be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf"
Nov 03 20:32:04.580692 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.580589    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/956c41e5-c95c-4dc4-8734-99bd8cecd1be-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf\" (UID: \"956c41e5-c95c-4dc4-8734-99bd8cecd1be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf"
Nov 03 20:32:04.580848 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.580737    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/956c41e5-c95c-4dc4-8734-99bd8cecd1be-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf\" (UID: \"956c41e5-c95c-4dc4-8734-99bd8cecd1be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf"
Nov 03 20:32:04.580917 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.580864    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb" podStartSLOduration=2.580848437 podStartE2EDuration="2.580848437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:04.555725468 +0000 UTC m=+3273.328018487" watchObservedRunningTime="2026-11-03 20:32:04.580848437 +0000 UTC m=+3273.353141456"
Nov 03 20:32:04.580992 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.580945    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/956c41e5-c95c-4dc4-8734-99bd8cecd1be-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf\" (UID: \"956c41e5-c95c-4dc4-8734-99bd8cecd1be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf"
Nov 03 20:32:04.581053 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.581026    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/956c41e5-c95c-4dc4-8734-99bd8cecd1be-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf\" (UID: \"956c41e5-c95c-4dc4-8734-99bd8cecd1be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf"
Nov 03 20:32:04.581105 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.581055    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7stdz\" (UniqueName: \"kubernetes.io/projected/956c41e5-c95c-4dc4-8734-99bd8cecd1be-kube-api-access-7stdz\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf\" (UID: \"956c41e5-c95c-4dc4-8734-99bd8cecd1be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf"
Nov 03 20:32:04.584442 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.584416    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/956c41e5-c95c-4dc4-8734-99bd8cecd1be-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf\" (UID: \"956c41e5-c95c-4dc4-8734-99bd8cecd1be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf"
Nov 03 20:32:04.584798 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.584705    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/956c41e5-c95c-4dc4-8734-99bd8cecd1be-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf\" (UID: \"956c41e5-c95c-4dc4-8734-99bd8cecd1be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf"
Nov 03 20:32:04.587762 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.587740    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/956c41e5-c95c-4dc4-8734-99bd8cecd1be-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf\" (UID: \"956c41e5-c95c-4dc4-8734-99bd8cecd1be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf"
Nov 03 20:32:04.588188 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.588154    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/956c41e5-c95c-4dc4-8734-99bd8cecd1be-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf\" (UID: \"956c41e5-c95c-4dc4-8734-99bd8cecd1be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf"
Nov 03 20:32:04.597255 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.589594    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/956c41e5-c95c-4dc4-8734-99bd8cecd1be-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf\" (UID: \"956c41e5-c95c-4dc4-8734-99bd8cecd1be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf"
Nov 03 20:32:04.619571 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.619392    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7stdz\" (UniqueName: \"kubernetes.io/projected/956c41e5-c95c-4dc4-8734-99bd8cecd1be-kube-api-access-7stdz\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf\" (UID: \"956c41e5-c95c-4dc4-8734-99bd8cecd1be\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf"
Nov 03 20:32:04.623911 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.623857    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh" podStartSLOduration=2.623824478 podStartE2EDuration="2.623824478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:04.619465026 +0000 UTC m=+3273.391758045" watchObservedRunningTime="2026-11-03 20:32:04.623824478 +0000 UTC m=+3273.396117498"
Nov 03 20:32:04.651802 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.651720    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46" podStartSLOduration=2.651705803 podStartE2EDuration="2.651705803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:04.643929822 +0000 UTC m=+3273.416222844" watchObservedRunningTime="2026-11-03 20:32:04.651705803 +0000 UTC m=+3273.423998821"
Nov 03 20:32:04.691777 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.691737    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-home\") pod \"simulator-w1-32e482-kserve-776bf6595d-qlczq\" (UID: \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"
Nov 03 20:32:04.691988 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.691824    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-tmp-dir\") pod \"simulator-w1-32e482-kserve-776bf6595d-qlczq\" (UID: \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"
Nov 03 20:32:04.691988 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.691914    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kcws\" (UniqueName: \"kubernetes.io/projected/8c68cd56-eb04-4f8e-806a-510cc9ec2644-kube-api-access-6kcws\") pod \"simulator-w1-32e482-kserve-776bf6595d-qlczq\" (UID: \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"
Nov 03 20:32:04.692667 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.692646    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-h8wps"]
Nov 03 20:32:04.692969 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.692941    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-tls-certs\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-fw6ml\" (UID: \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"
Nov 03 20:32:04.693046 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.693020    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-dshm\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-fw6ml\" (UID: \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"
Nov 03 20:32:04.693106 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.693054    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd6cq\" (UniqueName: \"kubernetes.io/projected/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-kube-api-access-xd6cq\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-fw6ml\" (UID: \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"
Nov 03 20:32:04.693162 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.693148    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-model-cache\") pod \"simulator-w1-32e482-kserve-776bf6595d-qlczq\" (UID: \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"
Nov 03 20:32:04.693276 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.693173    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c68cd56-eb04-4f8e-806a-510cc9ec2644-tls-certs\") pod \"simulator-w1-32e482-kserve-776bf6595d-qlczq\" (UID: \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"
Nov 03 20:32:04.693335 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.693289    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-tmp-dir\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-fw6ml\" (UID: \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"
Nov 03 20:32:04.693490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.693471    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-dshm\") pod \"simulator-w1-32e482-kserve-776bf6595d-qlczq\" (UID: \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"
Nov 03 20:32:04.693545 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.693525    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-model-cache\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-fw6ml\" (UID: \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"
Nov 03 20:32:04.695779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.695656    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-home\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-fw6ml\" (UID: \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"
Nov 03 20:32:04.698851 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.698833    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-h8wps"
Nov 03 20:32:04.708315 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.706228    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 03 20:32:04.708315 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.706403    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"kas-connection-checker-dockercfg-ph9pf\""
Nov 03 20:32:04.708315 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.706445    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 03 20:32:04.708315 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.706470    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9pxsv"]
Nov 03 20:32:04.708635 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.708431    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf"
Nov 03 20:32:04.752604 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.752570    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gwxd6"]
Nov 03 20:32:04.755586 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.753995    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9pxsv"
Nov 03 20:32:04.764852 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.764678    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-gwxd6"
Nov 03 20:32:04.775356 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.775319    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9pxsv"]
Nov 03 20:32:04.779441 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.779408    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-h8wps"]
Nov 03 20:32:04.790667 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.790638    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gwxd6"]
Nov 03 20:32:04.797342 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.797311    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd6cq\" (UniqueName: \"kubernetes.io/projected/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-kube-api-access-xd6cq\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-fw6ml\" (UID: \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"
Nov 03 20:32:04.797471 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.797384    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6hv2\" (UniqueName: \"kubernetes.io/projected/1fe14dc7-e65e-4115-a93a-04719aa38622-kube-api-access-q6hv2\") pod \"kas-connection-checker-685895c7c9-gwxd6\" (UID: \"1fe14dc7-e65e-4115-a93a-04719aa38622\") " pod="kube-system/kas-connection-checker-685895c7c9-gwxd6"
Nov 03 20:32:04.797471 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.797449    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-model-cache\") pod \"simulator-w1-32e482-kserve-776bf6595d-qlczq\" (UID: \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"
Nov 03 20:32:04.797551 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.797482    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c68cd56-eb04-4f8e-806a-510cc9ec2644-tls-certs\") pod \"simulator-w1-32e482-kserve-776bf6595d-qlczq\" (UID: \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"
Nov 03 20:32:04.797551 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.797515    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-tmp-dir\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-fw6ml\" (UID: \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"
Nov 03 20:32:04.797551 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.797538    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7p62\" (UniqueName: \"kubernetes.io/projected/585afe81-6315-4c26-b8fb-2caa9c6c121a-kube-api-access-p7p62\") pod \"kas-connection-checker-685895c7c9-h8wps\" (UID: \"585afe81-6315-4c26-b8fb-2caa9c6c121a\") " pod="kube-system/kas-connection-checker-685895c7c9-h8wps"
Nov 03 20:32:04.797729 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.797712    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-dshm\") pod \"simulator-w1-32e482-kserve-776bf6595d-qlczq\" (UID: \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"
Nov 03 20:32:04.797789 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.797750    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-model-cache\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-fw6ml\" (UID: \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"
Nov 03 20:32:04.798120 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.798103    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-home\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-fw6ml\" (UID: \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"
Nov 03 20:32:04.798177 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.798167    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-home\") pod \"simulator-w1-32e482-kserve-776bf6595d-qlczq\" (UID: \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"
Nov 03 20:32:04.798251 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.798228    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvksc\" (UniqueName: \"kubernetes.io/projected/4b60e6cd-0733-43f9-ac06-b7183f8ddbe6-kube-api-access-nvksc\") pod \"kas-connection-checker-685895c7c9-9pxsv\" (UID: \"4b60e6cd-0733-43f9-ac06-b7183f8ddbe6\") " pod="kube-system/kas-connection-checker-685895c7c9-9pxsv"
Nov 03 20:32:04.798287 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.798258    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-tmp-dir\") pod \"simulator-w1-32e482-kserve-776bf6595d-qlczq\" (UID: \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"
Nov 03 20:32:04.799913 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.799893    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-tmp-dir\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-fw6ml\" (UID: \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"
Nov 03 20:32:04.799913 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.799893    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-tmp-dir\") pod \"simulator-w1-32e482-kserve-776bf6595d-qlczq\" (UID: \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"
Nov 03 20:32:04.800138 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.799950    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kcws\" (UniqueName: \"kubernetes.io/projected/8c68cd56-eb04-4f8e-806a-510cc9ec2644-kube-api-access-6kcws\") pod \"simulator-w1-32e482-kserve-776bf6595d-qlczq\" (UID: \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"
Nov 03 20:32:04.800138 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.799981    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-tls-certs\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-fw6ml\" (UID: \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"
Nov 03 20:32:04.800138 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.800010    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-dshm\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-fw6ml\" (UID: \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"
Nov 03 20:32:04.805322 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.802412    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-dshm\") pod \"simulator-w1-32e482-kserve-776bf6595d-qlczq\" (UID: \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"
Nov 03 20:32:04.805322 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.802610    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-model-cache\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-fw6ml\" (UID: \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"
Nov 03 20:32:04.805322 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.802782    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-home\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-fw6ml\" (UID: \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"
Nov 03 20:32:04.805322 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.802949    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-home\") pod \"simulator-w1-32e482-kserve-776bf6595d-qlczq\" (UID: \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"
Nov 03 20:32:04.805322 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.803255    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-dshm\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-fw6ml\" (UID: \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"
Nov 03 20:32:04.805841 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.803485    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-model-cache\") pod \"simulator-w1-32e482-kserve-776bf6595d-qlczq\" (UID: \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"
Nov 03 20:32:04.807513 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.807491    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c68cd56-eb04-4f8e-806a-510cc9ec2644-tls-certs\") pod \"simulator-w1-32e482-kserve-776bf6595d-qlczq\" (UID: \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"
Nov 03 20:32:04.809299 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.809197    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-tls-certs\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-fw6ml\" (UID: \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"
Nov 03 20:32:04.840325 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.821685    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd6cq\" (UniqueName: \"kubernetes.io/projected/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-kube-api-access-xd6cq\") pod \"distinct-w2-2b5703-kserve-676bd58bd6-fw6ml\" (UID: \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\") " pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"
Nov 03 20:32:04.849074 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.848552    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/odh-model-controller-769875db56-w9ppp"]
Nov 03 20:32:04.849074 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.848792    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/odh-model-controller-769875db56-w9ppp" podUID="55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d" containerName="manager" containerID="cri-o://abba62fc47336c1b1f46f30566594de1ec1b62b992313b2f933a031d60410ffc" gracePeriod=10
Nov 03 20:32:04.855964 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.855347    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kcws\" (UniqueName: \"kubernetes.io/projected/8c68cd56-eb04-4f8e-806a-510cc9ec2644-kube-api-access-6kcws\") pod \"simulator-w1-32e482-kserve-776bf6595d-qlczq\" (UID: \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\") " pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"
Nov 03 20:32:04.901718 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.901679    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6hv2\" (UniqueName: \"kubernetes.io/projected/1fe14dc7-e65e-4115-a93a-04719aa38622-kube-api-access-q6hv2\") pod \"kas-connection-checker-685895c7c9-gwxd6\" (UID: \"1fe14dc7-e65e-4115-a93a-04719aa38622\") " pod="kube-system/kas-connection-checker-685895c7c9-gwxd6"
Nov 03 20:32:04.902404 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.902338    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7p62\" (UniqueName: \"kubernetes.io/projected/585afe81-6315-4c26-b8fb-2caa9c6c121a-kube-api-access-p7p62\") pod \"kas-connection-checker-685895c7c9-h8wps\" (UID: \"585afe81-6315-4c26-b8fb-2caa9c6c121a\") " pod="kube-system/kas-connection-checker-685895c7c9-h8wps"
Nov 03 20:32:04.903282 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.903254    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvksc\" (UniqueName: \"kubernetes.io/projected/4b60e6cd-0733-43f9-ac06-b7183f8ddbe6-kube-api-access-nvksc\") pod \"kas-connection-checker-685895c7c9-9pxsv\" (UID: \"4b60e6cd-0733-43f9-ac06-b7183f8ddbe6\") " pod="kube-system/kas-connection-checker-685895c7c9-9pxsv"
Nov 03 20:32:04.923098 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.923067    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7p62\" (UniqueName: \"kubernetes.io/projected/585afe81-6315-4c26-b8fb-2caa9c6c121a-kube-api-access-p7p62\") pod \"kas-connection-checker-685895c7c9-h8wps\" (UID: \"585afe81-6315-4c26-b8fb-2caa9c6c121a\") " pod="kube-system/kas-connection-checker-685895c7c9-h8wps"
Nov 03 20:32:04.924809 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.924781    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6hv2\" (UniqueName: \"kubernetes.io/projected/1fe14dc7-e65e-4115-a93a-04719aa38622-kube-api-access-q6hv2\") pod \"kas-connection-checker-685895c7c9-gwxd6\" (UID: \"1fe14dc7-e65e-4115-a93a-04719aa38622\") " pod="kube-system/kas-connection-checker-685895c7c9-gwxd6"
Nov 03 20:32:04.924920 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:04.924828    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvksc\" (UniqueName: \"kubernetes.io/projected/4b60e6cd-0733-43f9-ac06-b7183f8ddbe6-kube-api-access-nvksc\") pod \"kas-connection-checker-685895c7c9-9pxsv\" (UID: \"4b60e6cd-0733-43f9-ac06-b7183f8ddbe6\") " pod="kube-system/kas-connection-checker-685895c7c9-9pxsv"
Nov 03 20:32:05.037406 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:32:05.037371    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod956c41e5_c95c_4dc4_8734_99bd8cecd1be.slice/crio-4e21b43ebe2076b64d6492633562ef72be7397a114663358f6e95c90d5fae2f2 WatchSource:0}: Error finding container 4e21b43ebe2076b64d6492633562ef72be7397a114663358f6e95c90d5fae2f2: Status 404 returned error can't find the container with id 4e21b43ebe2076b64d6492633562ef72be7397a114663358f6e95c90d5fae2f2
Nov 03 20:32:05.042791 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.042760    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf"]
Nov 03 20:32:05.087776 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.087562    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-h8wps"
Nov 03 20:32:05.087987 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.087962    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"
Nov 03 20:32:05.089591 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.089354    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"
Nov 03 20:32:05.162093 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.161977    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9pxsv"
Nov 03 20:32:05.192321 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.187175    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-gwxd6"
Nov 03 20:32:05.249735 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.249689    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-w9ppp"
Nov 03 20:32:05.261099 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.261066    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"
Nov 03 20:32:05.326260 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.325489    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d-cert\" (UniqueName: \"kubernetes.io/secret/55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d-cert\") pod \"55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d\" (UID: \"55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d\") "
Nov 03 20:32:05.326260 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.325705    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d-kube-api-access-5vczd\" (UniqueName: \"kubernetes.io/projected/55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d-kube-api-access-5vczd\") pod \"55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d\" (UID: \"55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d\") "
Nov 03 20:32:05.326260 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.325924    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d-metrics-certs\") pod \"55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d\" (UID: \"55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d\") "
Nov 03 20:32:05.418386 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.418322    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d-metrics-certs" pod "55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d" (UID: "55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:32:05.420304 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.418711    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d-kube-api-access-5vczd" pod "55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d" (UID: "55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d"). InnerVolumeSpecName "kube-api-access-5vczd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:05.422356 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.422258    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d-cert" pod "55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d" (UID: "55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:32:05.426144 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.426114    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a28774-81b4-4051-9e4f-cb8fbe703942" path="/var/lib/kubelet/pods/07a28774-81b4-4051-9e4f-cb8fbe703942/volumes"
Nov 03 20:32:05.433686 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.433582    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:05.433686 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.433609    5591 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:05.433686 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.433625    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vczd\" (UniqueName: \"kubernetes.io/projected/55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d-kube-api-access-5vczd\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:05.509770 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.506494    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"
Nov 03 20:32:05.515269 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.514113    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-h8wps"]
Nov 03 20:32:05.541512 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.541458    5591 generic.go:356] "Generic (PLEG): container finished" podID="55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d" containerID="abba62fc47336c1b1f46f30566594de1ec1b62b992313b2f933a031d60410ffc" exitCode=0
Nov 03 20:32:05.541752 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.541727    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-w9ppp" event={"ID":"55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d","Type":"ContainerDied","Data":"abba62fc47336c1b1f46f30566594de1ec1b62b992313b2f933a031d60410ffc"}
Nov 03 20:32:05.541933 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.541915    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-w9ppp" event={"ID":"55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d","Type":"ContainerDied","Data":"6523f04611daf97cbb6737e5422a2736d5a7f4c915c97fc13d5d43e64abe58d6"}
Nov 03 20:32:05.542100 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.541770    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-w9ppp"
Nov 03 20:32:05.542269 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.542047    5591 scope.go:122] "RemoveContainer" containerID="abba62fc47336c1b1f46f30566594de1ec1b62b992313b2f933a031d60410ffc"
Nov 03 20:32:05.548899 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.548870    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6x2hv" event={"ID":"dc03a37b-a84b-4df5-9306-57b8e1ec8f7b","Type":"ContainerStarted","Data":"fb59a0c5634faee120414e032e5ff27d1ce0bc2420590898a2cd62d32ad7b251"}
Nov 03 20:32:05.549962 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.549915    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6x2hv"
Nov 03 20:32:05.557147 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.557124    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6x2hv"
Nov 03 20:32:05.558961 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.558934    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf" event={"ID":"956c41e5-c95c-4dc4-8734-99bd8cecd1be","Type":"ContainerStarted","Data":"4e21b43ebe2076b64d6492633562ef72be7397a114663358f6e95c90d5fae2f2"}
Nov 03 20:32:05.566533 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.564834    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-h8wps" event={"ID":"585afe81-6315-4c26-b8fb-2caa9c6c121a","Type":"ContainerStarted","Data":"d6c0f9197b49d83a901b0635d306abbab3215f6eb91395cd640db6ec50671e47"}
Nov 03 20:32:05.581060 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.581011    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/odh-model-controller-769875db56-w9ppp"]
Nov 03 20:32:05.587075 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.586975    5591 scope.go:122] "RemoveContainer" containerID="abba62fc47336c1b1f46f30566594de1ec1b62b992313b2f933a031d60410ffc"
Nov 03 20:32:05.589115 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:05.588925    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abba62fc47336c1b1f46f30566594de1ec1b62b992313b2f933a031d60410ffc\": container with ID starting with abba62fc47336c1b1f46f30566594de1ec1b62b992313b2f933a031d60410ffc not found: ID does not exist" containerID="abba62fc47336c1b1f46f30566594de1ec1b62b992313b2f933a031d60410ffc"
Nov 03 20:32:05.589115 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.588951    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abba62fc47336c1b1f46f30566594de1ec1b62b992313b2f933a031d60410ffc"} err="failed to get container status \"abba62fc47336c1b1f46f30566594de1ec1b62b992313b2f933a031d60410ffc\": rpc error: code = NotFound desc = could not find container \"abba62fc47336c1b1f46f30566594de1ec1b62b992313b2f933a031d60410ffc\": container with ID starting with abba62fc47336c1b1f46f30566594de1ec1b62b992313b2f933a031d60410ffc not found: ID does not exist"
Nov 03 20:32:05.597263 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.596848    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/odh-model-controller-769875db56-w9ppp"]
Nov 03 20:32:05.623578 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.623515    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6x2hv" podStartSLOduration=2.089507573 podStartE2EDuration="3.62349579s" totalImagesPullingTime="1.533988217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:05.613057523 +0000 UTC m=+3274.385350547" watchObservedRunningTime="2026-11-03 20:32:05.62349579 +0000 UTC m=+3274.395788809"
Nov 03 20:32:05.825868 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.825828    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"]
Nov 03 20:32:05.854293 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:32:05.854254    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c68cd56_eb04_4f8e_806a_510cc9ec2644.slice/crio-54122ecc923c054938b621c6d40090c2393dd899036c0a0c39878bb72ec141fa WatchSource:0}: Error finding container 54122ecc923c054938b621c6d40090c2393dd899036c0a0c39878bb72ec141fa: Status 404 returned error can't find the container with id 54122ecc923c054938b621c6d40090c2393dd899036c0a0c39878bb72ec141fa
Nov 03 20:32:05.886640 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.886456    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"]
Nov 03 20:32:05.950854 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.950784    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gwxd6"]
Nov 03 20:32:05.960513 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:05.960486    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9pxsv"]
Nov 03 20:32:06.090988 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:06.088639    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:32:06.090988 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:06.088693    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn" podUID="ea34ecc5-6c75-44b0-a28a-530668e79e47" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:32:06.582340 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:06.582276    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf" event={"ID":"956c41e5-c95c-4dc4-8734-99bd8cecd1be","Type":"ContainerStarted","Data":"99d4a3d1d58ac6b17c83c6c9855996f708f69d0a4304d25a7c8cd95004e853f9"}
Nov 03 20:32:06.585186 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:06.585154    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-gwxd6" event={"ID":"1fe14dc7-e65e-4115-a93a-04719aa38622","Type":"ContainerStarted","Data":"de0948623d716546bb4708a094400513c71a56bdac985ba2fa5c741067b71208"}
Nov 03 20:32:06.585379 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:06.585250    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-gwxd6" event={"ID":"1fe14dc7-e65e-4115-a93a-04719aa38622","Type":"ContainerStarted","Data":"06bb43f2ce5ebec90dfca0056284bb6c276315a83735cc091104aac003e86c78"}
Nov 03 20:32:06.588857 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:06.588824    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-h8wps" event={"ID":"585afe81-6315-4c26-b8fb-2caa9c6c121a","Type":"ContainerStarted","Data":"1b9afa016975fe43e13a30fe53dc62534853054cabeef99d06adac1c24dde77a"}
Nov 03 20:32:06.592804 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:06.592775    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq" event={"ID":"8c68cd56-eb04-4f8e-806a-510cc9ec2644","Type":"ContainerStarted","Data":"567ed7d5563fc18a5b8f47314ca0abb1c13bb535e34e7f81c908c7be1adecf02"}
Nov 03 20:32:06.592934 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:06.592816    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq" event={"ID":"8c68cd56-eb04-4f8e-806a-510cc9ec2644","Type":"ContainerStarted","Data":"54122ecc923c054938b621c6d40090c2393dd899036c0a0c39878bb72ec141fa"}
Nov 03 20:32:06.595631 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:06.595606    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml" event={"ID":"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068","Type":"ContainerStarted","Data":"5bb6854de3819ef662794d84b6fb0c8de6ed539fbb85cbef337ccd2b0aa7fcf8"}
Nov 03 20:32:06.595742 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:06.595643    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml" event={"ID":"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068","Type":"ContainerStarted","Data":"cf95d59efa66e5ce5464406e2b0de99225bfacc139eadb17365a156f2e72e620"}
Nov 03 20:32:06.599169 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:06.599145    5591 patch_prober.go:29] interesting pod/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:32:06.599408 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:06.599385    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:32:06.600477 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:06.600449    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9pxsv" event={"ID":"4b60e6cd-0733-43f9-ac06-b7183f8ddbe6","Type":"ContainerStarted","Data":"46230d83194775342882e7503a138e4b99753e30b333ee897f4a203a002efc4b"}
Nov 03 20:32:06.600572 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:06.600499    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9pxsv" event={"ID":"4b60e6cd-0733-43f9-ac06-b7183f8ddbe6","Type":"ContainerStarted","Data":"1e4b81c0b33588b9501cbb5998855000603de11e960c6203781717077455f497"}
Nov 03 20:32:06.608766 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:06.608692    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf" podStartSLOduration=3.084062872 podStartE2EDuration="3.608676861s" totalImagesPullingTime="524.613989ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:06.60471897 +0000 UTC m=+3275.377011989" watchObservedRunningTime="2026-11-03 20:32:06.608676861 +0000 UTC m=+3275.380969880"
Nov 03 20:32:06.694880 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:06.694815    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml" podStartSLOduration=3.6947956360000003 podStartE2EDuration="3.694795636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:06.690296583 +0000 UTC m=+3275.462589602" watchObservedRunningTime="2026-11-03 20:32:06.694795636 +0000 UTC m=+3275.467088655"
Nov 03 20:32:06.710737 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:06.710689    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf"
Nov 03 20:32:06.746701 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:06.746637    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq" podStartSLOduration=3.7466158480000002 podStartE2EDuration="3.746615848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:06.743405793 +0000 UTC m=+3275.515698813" watchObservedRunningTime="2026-11-03 20:32:06.746615848 +0000 UTC m=+3275.518908870"
Nov 03 20:32:06.811541 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:06.811481    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-gwxd6" podStartSLOduration=3.81146495 podStartE2EDuration="3.81146495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:06.805558316 +0000 UTC m=+3275.577851335" watchObservedRunningTime="2026-11-03 20:32:06.81146495 +0000 UTC m=+3275.583757959"
Nov 03 20:32:06.847791 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:06.847695    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-h8wps" podStartSLOduration=3.8476803410000002 podStartE2EDuration="3.847680341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:06.845012011 +0000 UTC m=+3275.617305030" watchObservedRunningTime="2026-11-03 20:32:06.847680341 +0000 UTC m=+3275.619973405"
Nov 03 20:32:06.878926 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:06.878851    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-9pxsv" podStartSLOduration=3.87882511 podStartE2EDuration="3.87882511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:06.87493778 +0000 UTC m=+3275.647230789" watchObservedRunningTime="2026-11-03 20:32:06.87882511 +0000 UTC m=+3275.651118129"
Nov 03 20:32:06.943229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:06.933309    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-kqcwg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:32:06.943229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:06.942696    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg" podUID="53641647-e68c-4a85-8d1a-fa903d2acdad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:32:07.088575 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:07.088535    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"
Nov 03 20:32:07.089631 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:07.089605    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"
Nov 03 20:32:07.354779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:07.354749    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d" path="/var/lib/kubelet/pods/55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d/volumes"
Nov 03 20:32:08.087250 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.083105    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:32:08.167733 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.167293    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:32:08.219661 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.219617    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3163904f-cc70-40d4-b196-563e2b900b29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3163904f-cc70-40d4-b196-563e2b900b29-istiod-ca-cert\") pod \"3163904f-cc70-40d4-b196-563e2b900b29\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") "
Nov 03 20:32:08.219867 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.219685    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-istio-envoy\") pod \"3163904f-cc70-40d4-b196-563e2b900b29\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") "
Nov 03 20:32:08.219945 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.219924    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3163904f-cc70-40d4-b196-563e2b900b29-kube-api-access-4f8ks\" (UniqueName: \"kubernetes.io/projected/3163904f-cc70-40d4-b196-563e2b900b29-kube-api-access-4f8ks\") pod \"3163904f-cc70-40d4-b196-563e2b900b29\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") "
Nov 03 20:32:08.220002 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.219988    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-workload-certs\") pod \"3163904f-cc70-40d4-b196-563e2b900b29\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") "
Nov 03 20:32:08.220077 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.220037    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3163904f-cc70-40d4-b196-563e2b900b29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3163904f-cc70-40d4-b196-563e2b900b29-istio-podinfo\") pod \"3163904f-cc70-40d4-b196-563e2b900b29\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") "
Nov 03 20:32:08.220291 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.220262    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-istio-data\") pod \"3163904f-cc70-40d4-b196-563e2b900b29\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") "
Nov 03 20:32:08.220714 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.220687    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3163904f-cc70-40d4-b196-563e2b900b29-istio-token\" (UniqueName: \"kubernetes.io/projected/3163904f-cc70-40d4-b196-563e2b900b29-istio-token\") pod \"3163904f-cc70-40d4-b196-563e2b900b29\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") "
Nov 03 20:32:08.220830 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.220741    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-credential-socket\") pod \"3163904f-cc70-40d4-b196-563e2b900b29\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") "
Nov 03 20:32:08.220892 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.220827    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-workload-socket\") pod \"3163904f-cc70-40d4-b196-563e2b900b29\" (UID: \"3163904f-cc70-40d4-b196-563e2b900b29\") "
Nov 03 20:32:08.222115 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.222084    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-workload-socket" pod "3163904f-cc70-40d4-b196-563e2b900b29" (UID: "3163904f-cc70-40d4-b196-563e2b900b29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:08.222330 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.222307    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-workload-certs" pod "3163904f-cc70-40d4-b196-563e2b900b29" (UID: "3163904f-cc70-40d4-b196-563e2b900b29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:08.222936 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.222901    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3163904f-cc70-40d4-b196-563e2b900b29-istiod-ca-cert" pod "3163904f-cc70-40d4-b196-563e2b900b29" (UID: "3163904f-cc70-40d4-b196-563e2b900b29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:08.225381 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.225345    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-istio-data" pod "3163904f-cc70-40d4-b196-563e2b900b29" (UID: "3163904f-cc70-40d4-b196-563e2b900b29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:08.225694 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.225674    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-credential-socket" pod "3163904f-cc70-40d4-b196-563e2b900b29" (UID: "3163904f-cc70-40d4-b196-563e2b900b29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:08.232036 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.231997    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3163904f-cc70-40d4-b196-563e2b900b29-istio-podinfo" pod "3163904f-cc70-40d4-b196-563e2b900b29" (UID: "3163904f-cc70-40d4-b196-563e2b900b29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:32:08.232511 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.232488    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3163904f-cc70-40d4-b196-563e2b900b29-istio-token" pod "3163904f-cc70-40d4-b196-563e2b900b29" (UID: "3163904f-cc70-40d4-b196-563e2b900b29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:08.232919 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.232893    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-istio-envoy" pod "3163904f-cc70-40d4-b196-563e2b900b29" (UID: "3163904f-cc70-40d4-b196-563e2b900b29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:08.242512 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.242288    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3163904f-cc70-40d4-b196-563e2b900b29-kube-api-access-4f8ks" pod "3163904f-cc70-40d4-b196-563e2b900b29" (UID: "3163904f-cc70-40d4-b196-563e2b900b29"). InnerVolumeSpecName "kube-api-access-4f8ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:08.323018 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.322980    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-credential-socket\") pod \"ea34ecc5-6c75-44b0-a28a-530668e79e47\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") "
Nov 03 20:32:08.323273 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.323044    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea34ecc5-6c75-44b0-a28a-530668e79e47-kube-api-access-22bvw\" (UniqueName: \"kubernetes.io/projected/ea34ecc5-6c75-44b0-a28a-530668e79e47-kube-api-access-22bvw\") pod \"ea34ecc5-6c75-44b0-a28a-530668e79e47\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") "
Nov 03 20:32:08.323273 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.323098    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-workload-socket\") pod \"ea34ecc5-6c75-44b0-a28a-530668e79e47\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") "
Nov 03 20:32:08.323431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.323351    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-credential-socket" pod "ea34ecc5-6c75-44b0-a28a-530668e79e47" (UID: "ea34ecc5-6c75-44b0-a28a-530668e79e47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:08.323999 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.323817    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-workload-socket" pod "ea34ecc5-6c75-44b0-a28a-530668e79e47" (UID: "ea34ecc5-6c75-44b0-a28a-530668e79e47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:08.324742 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.324383    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-envoy\") pod \"ea34ecc5-6c75-44b0-a28a-530668e79e47\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") "
Nov 03 20:32:08.324742 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.324519    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-workload-certs\") pod \"ea34ecc5-6c75-44b0-a28a-530668e79e47\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") "
Nov 03 20:32:08.324742 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.324606    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-data\") pod \"ea34ecc5-6c75-44b0-a28a-530668e79e47\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") "
Nov 03 20:32:08.324974 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.324774    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-workload-certs" pod "ea34ecc5-6c75-44b0-a28a-530668e79e47" (UID: "ea34ecc5-6c75-44b0-a28a-530668e79e47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:08.325028 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.324979    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-data" pod "ea34ecc5-6c75-44b0-a28a-530668e79e47" (UID: "ea34ecc5-6c75-44b0-a28a-530668e79e47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:08.325521 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.325141    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea34ecc5-6c75-44b0-a28a-530668e79e47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea34ecc5-6c75-44b0-a28a-530668e79e47-istiod-ca-cert\") pod \"ea34ecc5-6c75-44b0-a28a-530668e79e47\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") "
Nov 03 20:32:08.325521 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.325267    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-podinfo\") pod \"ea34ecc5-6c75-44b0-a28a-530668e79e47\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") "
Nov 03 20:32:08.325521 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.325364    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-token\" (UniqueName: \"kubernetes.io/projected/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-token\") pod \"ea34ecc5-6c75-44b0-a28a-530668e79e47\" (UID: \"ea34ecc5-6c75-44b0-a28a-530668e79e47\") "
Nov 03 20:32:08.325715 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.325521    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea34ecc5-6c75-44b0-a28a-530668e79e47-istiod-ca-cert" pod "ea34ecc5-6c75-44b0-a28a-530668e79e47" (UID: "ea34ecc5-6c75-44b0-a28a-530668e79e47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:08.326955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.326731    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:08.326955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.326760    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f8ks\" (UniqueName: \"kubernetes.io/projected/3163904f-cc70-40d4-b196-563e2b900b29-kube-api-access-4f8ks\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:08.326955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.326777    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea34ecc5-6c75-44b0-a28a-530668e79e47-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:08.326955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.326791    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:08.326955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.326805    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3163904f-cc70-40d4-b196-563e2b900b29-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:08.326955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.326818    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:08.326955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.326832    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3163904f-cc70-40d4-b196-563e2b900b29-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:08.326955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.326846    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:08.326955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.326859    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:08.326955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.326873    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:08.326955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.326886    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:08.326955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.326900    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3163904f-cc70-40d4-b196-563e2b900b29-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:08.326955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.326914    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3163904f-cc70-40d4-b196-563e2b900b29-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:08.326955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.326927    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:08.330387 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.330357    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-envoy" pod "ea34ecc5-6c75-44b0-a28a-530668e79e47" (UID: "ea34ecc5-6c75-44b0-a28a-530668e79e47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:08.331083 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.331024    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-podinfo" pod "ea34ecc5-6c75-44b0-a28a-530668e79e47" (UID: "ea34ecc5-6c75-44b0-a28a-530668e79e47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:32:08.331368 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.331337    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-token" pod "ea34ecc5-6c75-44b0-a28a-530668e79e47" (UID: "ea34ecc5-6c75-44b0-a28a-530668e79e47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:08.331450 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.331398    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea34ecc5-6c75-44b0-a28a-530668e79e47-kube-api-access-22bvw" pod "ea34ecc5-6c75-44b0-a28a-530668e79e47" (UID: "ea34ecc5-6c75-44b0-a28a-530668e79e47"). InnerVolumeSpecName "kube-api-access-22bvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:08.428755 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.428713    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:08.428755 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.428759    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:08.428984 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.428774    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22bvw\" (UniqueName: \"kubernetes.io/projected/ea34ecc5-6c75-44b0-a28a-530668e79e47-kube-api-access-22bvw\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:08.428984 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.428793    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea34ecc5-6c75-44b0-a28a-530668e79e47-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:08.435474 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.435449    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:32:08.529574 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.529533    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-credential-socket\") pod \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") "
Nov 03 20:32:08.529879 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.529591    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-workload-certs\") pod \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") "
Nov 03 20:32:08.529879 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.529653    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-token\" (UniqueName: \"kubernetes.io/projected/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-token\") pod \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") "
Nov 03 20:32:08.529879 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.529813    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-podinfo\") pod \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") "
Nov 03 20:32:08.529879 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.529852    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-data\") pod \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") "
Nov 03 20:32:08.530282 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.530258    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-workload-certs" pod "f51a1214-6af0-4ce3-b924-ca8d06c860fc" (UID: "f51a1214-6af0-4ce3-b924-ca8d06c860fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:08.530815 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.530494    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-credential-socket" pod "f51a1214-6af0-4ce3-b924-ca8d06c860fc" (UID: "f51a1214-6af0-4ce3-b924-ca8d06c860fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:08.532568 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.531396    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istiod-ca-cert\") pod \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") "
Nov 03 20:32:08.532568 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.531481    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-envoy\") pod \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") "
Nov 03 20:32:08.532568 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.531514    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f51a1214-6af0-4ce3-b924-ca8d06c860fc-kube-api-access-vprt2\" (UniqueName: \"kubernetes.io/projected/f51a1214-6af0-4ce3-b924-ca8d06c860fc-kube-api-access-vprt2\") pod \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") "
Nov 03 20:32:08.532568 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.531619    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-workload-socket\") pod \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\" (UID: \"f51a1214-6af0-4ce3-b924-ca8d06c860fc\") "
Nov 03 20:32:08.533251 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.532895    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istiod-ca-cert" pod "f51a1214-6af0-4ce3-b924-ca8d06c860fc" (UID: "f51a1214-6af0-4ce3-b924-ca8d06c860fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:08.533329 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.533252    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:08.533329 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.533274    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:08.533329 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.533289    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:08.533578 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.533545    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-workload-socket" pod "f51a1214-6af0-4ce3-b924-ca8d06c860fc" (UID: "f51a1214-6af0-4ce3-b924-ca8d06c860fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:08.533894 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.533873    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-data" pod "f51a1214-6af0-4ce3-b924-ca8d06c860fc" (UID: "f51a1214-6af0-4ce3-b924-ca8d06c860fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:08.535002 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.534943    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-token" pod "f51a1214-6af0-4ce3-b924-ca8d06c860fc" (UID: "f51a1214-6af0-4ce3-b924-ca8d06c860fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:08.542955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.542919    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51a1214-6af0-4ce3-b924-ca8d06c860fc-kube-api-access-vprt2" pod "f51a1214-6af0-4ce3-b924-ca8d06c860fc" (UID: "f51a1214-6af0-4ce3-b924-ca8d06c860fc"). InnerVolumeSpecName "kube-api-access-vprt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:08.543166 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.543136    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-podinfo" pod "f51a1214-6af0-4ce3-b924-ca8d06c860fc" (UID: "f51a1214-6af0-4ce3-b924-ca8d06c860fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:32:08.545078 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.545046    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-envoy" pod "f51a1214-6af0-4ce3-b924-ca8d06c860fc" (UID: "f51a1214-6af0-4ce3-b924-ca8d06c860fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:08.624853 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.624768    5591 generic.go:356] "Generic (PLEG): container finished" podID="ea34ecc5-6c75-44b0-a28a-530668e79e47" containerID="9bc089cd5e57f1afac4279e59f62b6e8f7bc7cc4671af1bc3079c826e3dcb246" exitCode=0
Nov 03 20:32:08.624853 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.624825    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"
Nov 03 20:32:08.625083 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.624859    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn" event={"ID":"ea34ecc5-6c75-44b0-a28a-530668e79e47","Type":"ContainerDied","Data":"9bc089cd5e57f1afac4279e59f62b6e8f7bc7cc4671af1bc3079c826e3dcb246"}
Nov 03 20:32:08.625083 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.624918    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn" event={"ID":"ea34ecc5-6c75-44b0-a28a-530668e79e47","Type":"ContainerDied","Data":"d224895b6f5dba7bb321ee79639269aedc45aa9178b483fb282a0304409fb174"}
Nov 03 20:32:08.625083 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.624940    5591 scope.go:122] "RemoveContainer" containerID="9bc089cd5e57f1afac4279e59f62b6e8f7bc7cc4671af1bc3079c826e3dcb246"
Nov 03 20:32:08.626984 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.626947    5591 generic.go:356] "Generic (PLEG): container finished" podID="3163904f-cc70-40d4-b196-563e2b900b29" containerID="c682b4964142f5e0b9b5103c615b3529905b71dd682fccfc7e7593ab45717cf3" exitCode=0
Nov 03 20:32:08.627108 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.627077    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" event={"ID":"3163904f-cc70-40d4-b196-563e2b900b29","Type":"ContainerDied","Data":"c682b4964142f5e0b9b5103c615b3529905b71dd682fccfc7e7593ab45717cf3"}
Nov 03 20:32:08.627169 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.627122    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5" event={"ID":"3163904f-cc70-40d4-b196-563e2b900b29","Type":"ContainerDied","Data":"a294e8820faaecc04182a50dd6fb22f9e8d21f849c4f8590ad1f36343be7f43d"}
Nov 03 20:32:08.627257 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.627194    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"
Nov 03 20:32:08.629551 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.629430    5591 generic.go:356] "Generic (PLEG): container finished" podID="f51a1214-6af0-4ce3-b924-ca8d06c860fc" containerID="93fa05cccc54e0d6a2772a1eb9120569c678c347d5d5600b78af40654c5b06b9" exitCode=0
Nov 03 20:32:08.629551 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.629545    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"
Nov 03 20:32:08.630763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.630736    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s" event={"ID":"f51a1214-6af0-4ce3-b924-ca8d06c860fc","Type":"ContainerDied","Data":"93fa05cccc54e0d6a2772a1eb9120569c678c347d5d5600b78af40654c5b06b9"}
Nov 03 20:32:08.630901 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.630795    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s" event={"ID":"f51a1214-6af0-4ce3-b924-ca8d06c860fc","Type":"ContainerDied","Data":"0d55072d21cabe3937c04ac8cea92d1b6d2ee2f84f41bac3bdc2d6985e22fa62"}
Nov 03 20:32:08.636409 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.636385    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:08.636524 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.636414    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:08.636524 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.636430    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:08.636524 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.636444    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:08.636524 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.636458    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vprt2\" (UniqueName: \"kubernetes.io/projected/f51a1214-6af0-4ce3-b924-ca8d06c860fc-kube-api-access-vprt2\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:08.636524 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.636473    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51a1214-6af0-4ce3-b924-ca8d06c860fc-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:08.641124 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.641100    5591 scope.go:122] "RemoveContainer" containerID="9bc089cd5e57f1afac4279e59f62b6e8f7bc7cc4671af1bc3079c826e3dcb246"
Nov 03 20:32:08.641530 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:08.641502    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bc089cd5e57f1afac4279e59f62b6e8f7bc7cc4671af1bc3079c826e3dcb246\": container with ID starting with 9bc089cd5e57f1afac4279e59f62b6e8f7bc7cc4671af1bc3079c826e3dcb246 not found: ID does not exist" containerID="9bc089cd5e57f1afac4279e59f62b6e8f7bc7cc4671af1bc3079c826e3dcb246"
Nov 03 20:32:08.641613 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.641542    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bc089cd5e57f1afac4279e59f62b6e8f7bc7cc4671af1bc3079c826e3dcb246"} err="failed to get container status \"9bc089cd5e57f1afac4279e59f62b6e8f7bc7cc4671af1bc3079c826e3dcb246\": rpc error: code = NotFound desc = could not find container \"9bc089cd5e57f1afac4279e59f62b6e8f7bc7cc4671af1bc3079c826e3dcb246\": container with ID starting with 9bc089cd5e57f1afac4279e59f62b6e8f7bc7cc4671af1bc3079c826e3dcb246 not found: ID does not exist"
Nov 03 20:32:08.641613 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.641563    5591 scope.go:122] "RemoveContainer" containerID="c682b4964142f5e0b9b5103c615b3529905b71dd682fccfc7e7593ab45717cf3"
Nov 03 20:32:08.655365 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.655345    5591 scope.go:122] "RemoveContainer" containerID="c682b4964142f5e0b9b5103c615b3529905b71dd682fccfc7e7593ab45717cf3"
Nov 03 20:32:08.655696 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:08.655674    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c682b4964142f5e0b9b5103c615b3529905b71dd682fccfc7e7593ab45717cf3\": container with ID starting with c682b4964142f5e0b9b5103c615b3529905b71dd682fccfc7e7593ab45717cf3 not found: ID does not exist" containerID="c682b4964142f5e0b9b5103c615b3529905b71dd682fccfc7e7593ab45717cf3"
Nov 03 20:32:08.655807 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.655705    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c682b4964142f5e0b9b5103c615b3529905b71dd682fccfc7e7593ab45717cf3"} err="failed to get container status \"c682b4964142f5e0b9b5103c615b3529905b71dd682fccfc7e7593ab45717cf3\": rpc error: code = NotFound desc = could not find container \"c682b4964142f5e0b9b5103c615b3529905b71dd682fccfc7e7593ab45717cf3\": container with ID starting with c682b4964142f5e0b9b5103c615b3529905b71dd682fccfc7e7593ab45717cf3 not found: ID does not exist"
Nov 03 20:32:08.655807 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.655728    5591 scope.go:122] "RemoveContainer" containerID="93fa05cccc54e0d6a2772a1eb9120569c678c347d5d5600b78af40654c5b06b9"
Nov 03 20:32:08.667599 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.667576    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"]
Nov 03 20:32:08.668886 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.668868    5591 scope.go:122] "RemoveContainer" containerID="93fa05cccc54e0d6a2772a1eb9120569c678c347d5d5600b78af40654c5b06b9"
Nov 03 20:32:08.669267 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:08.669188    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93fa05cccc54e0d6a2772a1eb9120569c678c347d5d5600b78af40654c5b06b9\": container with ID starting with 93fa05cccc54e0d6a2772a1eb9120569c678c347d5d5600b78af40654c5b06b9 not found: ID does not exist" containerID="93fa05cccc54e0d6a2772a1eb9120569c678c347d5d5600b78af40654c5b06b9"
Nov 03 20:32:08.669347 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.669277    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93fa05cccc54e0d6a2772a1eb9120569c678c347d5d5600b78af40654c5b06b9"} err="failed to get container status \"93fa05cccc54e0d6a2772a1eb9120569c678c347d5d5600b78af40654c5b06b9\": rpc error: code = NotFound desc = could not find container \"93fa05cccc54e0d6a2772a1eb9120569c678c347d5d5600b78af40654c5b06b9\": container with ID starting with 93fa05cccc54e0d6a2772a1eb9120569c678c347d5d5600b78af40654c5b06b9 not found: ID does not exist"
Nov 03 20:32:08.677546 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.677518    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-qnrd5"]
Nov 03 20:32:08.692560 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.692531    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"]
Nov 03 20:32:08.705196 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.705161    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-m8r2s"]
Nov 03 20:32:08.728838 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.728809    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"]
Nov 03 20:32:08.734779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:08.734754    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e7b70d-openshift-default-77f4dd475c-xdzdn"]
Nov 03 20:32:09.365043 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.364988    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3163904f-cc70-40d4-b196-563e2b900b29" path="/var/lib/kubelet/pods/3163904f-cc70-40d4-b196-563e2b900b29/volumes"
Nov 03 20:32:09.366361 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.366335    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea34ecc5-6c75-44b0-a28a-530668e79e47" path="/var/lib/kubelet/pods/ea34ecc5-6c75-44b0-a28a-530668e79e47/volumes"
Nov 03 20:32:09.367289 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.367268    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f51a1214-6af0-4ce3-b924-ca8d06c860fc" path="/var/lib/kubelet/pods/f51a1214-6af0-4ce3-b924-ca8d06c860fc/volumes"
Nov 03 20:32:09.512586 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.512503    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb" podUID="a97c5397-3fc1-46f5-9627-9681bcab980b" containerName="payload-pre-processing" containerID="cri-o://a4704e73b5cf7d9da8837491c2fb4d79d616f1cb76f076a8f861dbb1a8af690f" gracePeriod=25
Nov 03 20:32:09.618617 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.618588    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:32:09.643026 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.642990    5591 generic.go:356] "Generic (PLEG): container finished" podID="53641647-e68c-4a85-8d1a-fa903d2acdad" containerID="d3f75afa34deb64b495136125a529cffcb613a97568ccb7074588bfb377923ec" exitCode=0
Nov 03 20:32:09.643243 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.643047    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"
Nov 03 20:32:09.643243 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.643068    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg" event={"ID":"53641647-e68c-4a85-8d1a-fa903d2acdad","Type":"ContainerDied","Data":"d3f75afa34deb64b495136125a529cffcb613a97568ccb7074588bfb377923ec"}
Nov 03 20:32:09.643243 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.643131    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg" event={"ID":"53641647-e68c-4a85-8d1a-fa903d2acdad","Type":"ContainerDied","Data":"2cbe803389ab3ce699eba2e303e5d556472b064359477c28bfba4c0593a6179b"}
Nov 03 20:32:09.643243 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.643162    5591 scope.go:122] "RemoveContainer" containerID="d3f75afa34deb64b495136125a529cffcb613a97568ccb7074588bfb377923ec"
Nov 03 20:32:09.651373 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.651254    5591 generic.go:356] "Generic (PLEG): container finished" podID="a97c5397-3fc1-46f5-9627-9681bcab980b" containerID="a4704e73b5cf7d9da8837491c2fb4d79d616f1cb76f076a8f861dbb1a8af690f" exitCode=0
Nov 03 20:32:09.651373 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.651305    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb" event={"ID":"a97c5397-3fc1-46f5-9627-9681bcab980b","Type":"ContainerDied","Data":"a4704e73b5cf7d9da8837491c2fb4d79d616f1cb76f076a8f861dbb1a8af690f"}
Nov 03 20:32:09.665776 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.665439    5591 scope.go:122] "RemoveContainer" containerID="d3f75afa34deb64b495136125a529cffcb613a97568ccb7074588bfb377923ec"
Nov 03 20:32:09.668341 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:09.668310    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f75afa34deb64b495136125a529cffcb613a97568ccb7074588bfb377923ec\": container with ID starting with d3f75afa34deb64b495136125a529cffcb613a97568ccb7074588bfb377923ec not found: ID does not exist" containerID="d3f75afa34deb64b495136125a529cffcb613a97568ccb7074588bfb377923ec"
Nov 03 20:32:09.668473 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.668350    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f75afa34deb64b495136125a529cffcb613a97568ccb7074588bfb377923ec"} err="failed to get container status \"d3f75afa34deb64b495136125a529cffcb613a97568ccb7074588bfb377923ec\": rpc error: code = NotFound desc = could not find container \"d3f75afa34deb64b495136125a529cffcb613a97568ccb7074588bfb377923ec\": container with ID starting with d3f75afa34deb64b495136125a529cffcb613a97568ccb7074588bfb377923ec not found: ID does not exist"
Nov 03 20:32:09.764411 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.764372    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53641647-e68c-4a85-8d1a-fa903d2acdad-kube-api-access-pv5zr\" (UniqueName: \"kubernetes.io/projected/53641647-e68c-4a85-8d1a-fa903d2acdad-kube-api-access-pv5zr\") pod \"53641647-e68c-4a85-8d1a-fa903d2acdad\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") "
Nov 03 20:32:09.764589 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.764511    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53641647-e68c-4a85-8d1a-fa903d2acdad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53641647-e68c-4a85-8d1a-fa903d2acdad-istiod-ca-cert\") pod \"53641647-e68c-4a85-8d1a-fa903d2acdad\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") "
Nov 03 20:32:09.764702 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.764601    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-workload-certs\") pod \"53641647-e68c-4a85-8d1a-fa903d2acdad\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") "
Nov 03 20:32:09.765095 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.764810    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-credential-socket\") pod \"53641647-e68c-4a85-8d1a-fa903d2acdad\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") "
Nov 03 20:32:09.765095 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.764868    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-podinfo\") pod \"53641647-e68c-4a85-8d1a-fa903d2acdad\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") "
Nov 03 20:32:09.765095 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.764950    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-token\" (UniqueName: \"kubernetes.io/projected/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-token\") pod \"53641647-e68c-4a85-8d1a-fa903d2acdad\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") "
Nov 03 20:32:09.765095 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.764992    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-data\") pod \"53641647-e68c-4a85-8d1a-fa903d2acdad\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") "
Nov 03 20:32:09.765420 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.764973    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-workload-certs" pod "53641647-e68c-4a85-8d1a-fa903d2acdad" (UID: "53641647-e68c-4a85-8d1a-fa903d2acdad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:09.765420 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.765053    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-credential-socket" pod "53641647-e68c-4a85-8d1a-fa903d2acdad" (UID: "53641647-e68c-4a85-8d1a-fa903d2acdad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:09.765420 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.765315    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-data" pod "53641647-e68c-4a85-8d1a-fa903d2acdad" (UID: "53641647-e68c-4a85-8d1a-fa903d2acdad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:09.765681 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.765653    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-workload-socket\") pod \"53641647-e68c-4a85-8d1a-fa903d2acdad\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") "
Nov 03 20:32:09.765837 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.765719    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-envoy\") pod \"53641647-e68c-4a85-8d1a-fa903d2acdad\" (UID: \"53641647-e68c-4a85-8d1a-fa903d2acdad\") "
Nov 03 20:32:09.766324 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.766280    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-workload-socket" pod "53641647-e68c-4a85-8d1a-fa903d2acdad" (UID: "53641647-e68c-4a85-8d1a-fa903d2acdad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:09.766483 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.766444    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53641647-e68c-4a85-8d1a-fa903d2acdad-istiod-ca-cert" pod "53641647-e68c-4a85-8d1a-fa903d2acdad" (UID: "53641647-e68c-4a85-8d1a-fa903d2acdad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:09.797345 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.797297    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-envoy" pod "53641647-e68c-4a85-8d1a-fa903d2acdad" (UID: "53641647-e68c-4a85-8d1a-fa903d2acdad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:09.798351 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.798306    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53641647-e68c-4a85-8d1a-fa903d2acdad-kube-api-access-pv5zr" pod "53641647-e68c-4a85-8d1a-fa903d2acdad" (UID: "53641647-e68c-4a85-8d1a-fa903d2acdad"). InnerVolumeSpecName "kube-api-access-pv5zr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:09.798801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.798721    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.798801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.798740    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.798801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.798753    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53641647-e68c-4a85-8d1a-fa903d2acdad-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.798801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.798761    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.798801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.798771    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.800169 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.800115    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-podinfo" pod "53641647-e68c-4a85-8d1a-fa903d2acdad" (UID: "53641647-e68c-4a85-8d1a-fa903d2acdad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:32:09.801102 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.801060    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-token" pod "53641647-e68c-4a85-8d1a-fa903d2acdad" (UID: "53641647-e68c-4a85-8d1a-fa903d2acdad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:09.901289 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.901253    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv5zr\" (UniqueName: \"kubernetes.io/projected/53641647-e68c-4a85-8d1a-fa903d2acdad-kube-api-access-pv5zr\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.901289 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.901290    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.901495 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.901302    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.901495 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.901316    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53641647-e68c-4a85-8d1a-fa903d2acdad-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:09.909615 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.909586    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb"
Nov 03 20:32:09.985187 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.985145    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"]
Nov 03 20:32:09.995837 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:09.995799    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-kqcwg"]
Nov 03 20:32:10.002336 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:10.002304    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a97c5397-3fc1-46f5-9627-9681bcab980b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a97c5397-3fc1-46f5-9627-9681bcab980b-plugins-config-volume\") pod \"a97c5397-3fc1-46f5-9627-9681bcab980b\" (UID: \"a97c5397-3fc1-46f5-9627-9681bcab980b\") "
Nov 03 20:32:10.002567 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:10.002550    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a97c5397-3fc1-46f5-9627-9681bcab980b-kube-api-access-852jw\" (UniqueName: \"kubernetes.io/projected/a97c5397-3fc1-46f5-9627-9681bcab980b-kube-api-access-852jw\") pod \"a97c5397-3fc1-46f5-9627-9681bcab980b\" (UID: \"a97c5397-3fc1-46f5-9627-9681bcab980b\") "
Nov 03 20:32:10.002793 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:10.002765    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a97c5397-3fc1-46f5-9627-9681bcab980b-plugins-config-volume" pod "a97c5397-3fc1-46f5-9627-9681bcab980b" (UID: "a97c5397-3fc1-46f5-9627-9681bcab980b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:32:10.003416 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:10.003392    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a97c5397-3fc1-46f5-9627-9681bcab980b-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:10.004885 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:10.004863    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a97c5397-3fc1-46f5-9627-9681bcab980b-kube-api-access-852jw" pod "a97c5397-3fc1-46f5-9627-9681bcab980b" (UID: "a97c5397-3fc1-46f5-9627-9681bcab980b"). InnerVolumeSpecName "kube-api-access-852jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:10.105363 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:10.105320    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-852jw\" (UniqueName: \"kubernetes.io/projected/a97c5397-3fc1-46f5-9627-9681bcab980b-kube-api-access-852jw\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:10.666048 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:10.666006    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb" event={"ID":"a97c5397-3fc1-46f5-9627-9681bcab980b","Type":"ContainerDied","Data":"5976d8eeceef17fb1d8c88cdb5364793395fc18c4a6c9a99a27176e3fee67590"}
Nov 03 20:32:10.666599 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:10.666075    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb"
Nov 03 20:32:10.666599 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:10.666087    5591 scope.go:122] "RemoveContainer" containerID="a4704e73b5cf7d9da8837491c2fb4d79d616f1cb76f076a8f861dbb1a8af690f"
Nov 03 20:32:10.707599 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:10.707561    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb"]
Nov 03 20:32:10.713125 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:10.713103    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-32e482-5799655ddc-7pflb"]
Nov 03 20:32:11.358336 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:11.358302    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53641647-e68c-4a85-8d1a-fa903d2acdad" path="/var/lib/kubelet/pods/53641647-e68c-4a85-8d1a-fa903d2acdad/volumes"
Nov 03 20:32:11.359008 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:11.358983    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a97c5397-3fc1-46f5-9627-9681bcab980b" path="/var/lib/kubelet/pods/a97c5397-3fc1-46f5-9627-9681bcab980b/volumes"
Nov 03 20:32:11.675558 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:11.675521    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr" event={"ID":"d5cf239f-c014-4bfe-9479-88a51dbccff3","Type":"ContainerStarted","Data":"741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3"}
Nov 03 20:32:11.675558 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:11.675559    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr" podUID="d5cf239f-c014-4bfe-9479-88a51dbccff3" containerName="postgres" containerID="cri-o://741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3" gracePeriod=30
Nov 03 20:32:11.676039 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:11.675605    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr"
Nov 03 20:32:11.701402 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:11.701342    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr" podStartSLOduration=1.5049265950000001 podStartE2EDuration="9.701327806s" totalImagesPullingTime="8.196401211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:11.69689644 +0000 UTC m=+3280.469189460" watchObservedRunningTime="2026-11-03 20:32:11.701327806 +0000 UTC m=+3280.473620823"
Nov 03 20:32:13.261634 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:13.261594    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"
Nov 03 20:32:13.275314 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:13.275285    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"
Nov 03 20:32:13.506816 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:13.506786    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"
Nov 03 20:32:13.524031 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:13.524000    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"
Nov 03 20:32:13.710771 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:13.710737    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"
Nov 03 20:32:13.711671 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:13.711648    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"
Nov 03 20:32:14.711751 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:14.711707    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf"
Nov 03 20:32:14.724677 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:14.724649    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf"
Nov 03 20:32:15.088867 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:15.088819    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"
Nov 03 20:32:15.091409 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:15.090754    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"
Nov 03 20:32:15.103617 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:15.103585    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"
Nov 03 20:32:15.104623 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:15.104600    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"
Nov 03 20:32:15.722926 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:15.722893    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf"
Nov 03 20:32:15.730517 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:15.730488    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"
Nov 03 20:32:15.734791 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:15.734771    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"
Nov 03 20:32:16.676729 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:16.676679    5591 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3" cmd=["/usr/libexec/check-container"]
Nov 03 20:32:16.676917 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:16.676746    5591 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3" cmd=["/usr/libexec/check-container"]
Nov 03 20:32:16.676917 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:16.676762    5591 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3" cmd=["/usr/libexec/check-container"]
Nov 03 20:32:16.676917 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:16.676776    5591 prober.go:106] "Probe errored" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr" podUID="d5cf239f-c014-4bfe-9479-88a51dbccff3" containerName="postgres" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled"
Nov 03 20:32:21.676343 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:21.676290    5591 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3" cmd=["/usr/libexec/check-container"]
Nov 03 20:32:21.676939 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:21.676359    5591 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3" cmd=["/usr/libexec/check-container"]
Nov 03 20:32:21.676939 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:21.676384    5591 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3" cmd=["/usr/libexec/check-container"]
Nov 03 20:32:21.676939 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:21.676407    5591 prober.go:106] "Probe errored" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr" podUID="d5cf239f-c014-4bfe-9479-88a51dbccff3" containerName="postgres" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled"
Nov 03 20:32:26.676730 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:26.676670    5591 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3" cmd=["/usr/libexec/check-container"]
Nov 03 20:32:26.677387 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:26.676740    5591 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3" cmd=["/usr/libexec/check-container"]
Nov 03 20:32:26.677387 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:26.676761    5591 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3" cmd=["/usr/libexec/check-container"]
Nov 03 20:32:26.677387 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:26.676779    5591 prober.go:106] "Probe errored" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr" podUID="d5cf239f-c014-4bfe-9479-88a51dbccff3" containerName="postgres" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled"
Nov 03 20:32:28.736950 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:28.736910    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:32:31.676177 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:31.676127    5591 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3" cmd=["/usr/libexec/check-container"]
Nov 03 20:32:31.676177 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:31.676181    5591 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3" cmd=["/usr/libexec/check-container"]
Nov 03 20:32:31.676674 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:31.676197    5591 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3" cmd=["/usr/libexec/check-container"]
Nov 03 20:32:31.676674 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:31.676250    5591 prober.go:106] "Probe errored" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr" podUID="d5cf239f-c014-4bfe-9479-88a51dbccff3" containerName="postgres" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled"
Nov 03 20:32:33.998223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:33.998185    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 20:32:33.999697 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:33.999670    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 20:32:36.676352 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:36.676305    5591 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3" cmd=["/usr/libexec/check-container"]
Nov 03 20:32:36.676890 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:36.676360    5591 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3" cmd=["/usr/libexec/check-container"]
Nov 03 20:32:36.676890 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:36.676377    5591 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3" cmd=["/usr/libexec/check-container"]
Nov 03 20:32:36.676890 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:36.676391    5591 prober.go:106] "Probe errored" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr" podUID="d5cf239f-c014-4bfe-9479-88a51dbccff3" containerName="postgres" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled"
Nov 03 20:32:41.685538 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:41.685482    5591 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3" cmd=["/usr/libexec/check-container"]
Nov 03 20:32:41.686063 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:41.685557    5591 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3" cmd=["/usr/libexec/check-container"]
Nov 03 20:32:41.686063 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:41.685585    5591 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3" cmd=["/usr/libexec/check-container"]
Nov 03 20:32:41.686063 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:41.685605    5591 prober.go:106] "Probe errored" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr" podUID="d5cf239f-c014-4bfe-9479-88a51dbccff3" containerName="postgres" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled"
Nov 03 20:32:41.948195 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:41.948133    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr"
Nov 03 20:32:41.987328 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:41.987297    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58"]
Nov 03 20:32:41.989411 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:41.989381    5591 cpu_manager.go:414] "containerMap: removing container" podUID="a97c5397-3fc1-46f5-9627-9681bcab980b" containerName="payload-pre-processing"
Nov 03 20:32:41.989411 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:41.989410    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a97c5397-3fc1-46f5-9627-9681bcab980b" containerName="payload-pre-processing"
Nov 03 20:32:41.989641 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:41.989439    5591 cpu_manager.go:414] "containerMap: removing container" podUID="55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d" containerName="manager"
Nov 03 20:32:41.989641 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:41.989449    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d" containerName="manager"
Nov 03 20:32:41.989641 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:41.989467    5591 cpu_manager.go:414] "containerMap: removing container" podUID="f51a1214-6af0-4ce3-b924-ca8d06c860fc" containerName="istio-proxy"
Nov 03 20:32:41.989641 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:41.989474    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f51a1214-6af0-4ce3-b924-ca8d06c860fc" containerName="istio-proxy"
Nov 03 20:32:41.989641 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:41.989493    5591 cpu_manager.go:414] "containerMap: removing container" podUID="ea34ecc5-6c75-44b0-a28a-530668e79e47" containerName="istio-proxy"
Nov 03 20:32:41.989641 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:41.989500    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea34ecc5-6c75-44b0-a28a-530668e79e47" containerName="istio-proxy"
Nov 03 20:32:41.989641 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:41.989543    5591 cpu_manager.go:414] "containerMap: removing container" podUID="d5cf239f-c014-4bfe-9479-88a51dbccff3" containerName="postgres"
Nov 03 20:32:41.989641 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:41.989550    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5cf239f-c014-4bfe-9479-88a51dbccff3" containerName="postgres"
Nov 03 20:32:41.989641 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:41.989578    5591 cpu_manager.go:414] "containerMap: removing container" podUID="53641647-e68c-4a85-8d1a-fa903d2acdad" containerName="istio-proxy"
Nov 03 20:32:41.989641 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:41.989586    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53641647-e68c-4a85-8d1a-fa903d2acdad" containerName="istio-proxy"
Nov 03 20:32:41.989641 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:41.989607    5591 cpu_manager.go:414] "containerMap: removing container" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy"
Nov 03 20:32:41.989641 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:41.989614    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy"
Nov 03 20:32:41.990104 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:41.989843    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5cf239f-c014-4bfe-9479-88a51dbccff3" containerName="postgres"
Nov 03 20:32:41.990104 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:41.989865    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="53641647-e68c-4a85-8d1a-fa903d2acdad" containerName="istio-proxy"
Nov 03 20:32:41.990104 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:41.989878    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3163904f-cc70-40d4-b196-563e2b900b29" containerName="istio-proxy"
Nov 03 20:32:41.990104 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:41.989897    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="a97c5397-3fc1-46f5-9627-9681bcab980b" containerName="payload-pre-processing"
Nov 03 20:32:41.990104 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:41.989910    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="55ccd33d-4c5c-4d9b-8a54-e1ca4b48b24d" containerName="manager"
Nov 03 20:32:41.990104 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:41.989923    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f51a1214-6af0-4ce3-b924-ca8d06c860fc" containerName="istio-proxy"
Nov 03 20:32:41.990104 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:41.989935    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea34ecc5-6c75-44b0-a28a-530668e79e47" containerName="istio-proxy"
Nov 03 20:32:41.994893 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:41.994864    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58"
Nov 03 20:32:42.000338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.000228    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\""
Nov 03 20:32:42.000338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.000334    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24"]
Nov 03 20:32:42.000774 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.000739    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:32:42.002460 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.002435    5591 generic.go:356] "Generic (PLEG): container finished" podID="d5cf239f-c014-4bfe-9479-88a51dbccff3" containerID="741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3" exitCode=137
Nov 03 20:32:42.002811 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.002693    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr"
Nov 03 20:32:42.007146 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.007079    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr" event={"ID":"d5cf239f-c014-4bfe-9479-88a51dbccff3","Type":"ContainerDied","Data":"741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3"}
Nov 03 20:32:42.007146 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.007122    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr" event={"ID":"d5cf239f-c014-4bfe-9479-88a51dbccff3","Type":"ContainerDied","Data":"b7cdd6f944f6dd926826b9401e050374d874e37f7074a2c86b352cb400deb238"}
Nov 03 20:32:42.007485 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.007150    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"]
Nov 03 20:32:42.011330 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.011308    5591 scope.go:122] "RemoveContainer" containerID="741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3"
Nov 03 20:32:42.016013 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.015990    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24"
Nov 03 20:32:42.024782 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.024736    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c57db574-5845n"]
Nov 03 20:32:42.024955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.024903    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"
Nov 03 20:32:42.033860 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.033036    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58"]
Nov 03 20:32:42.034019 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.033141    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:32:42.034514 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.034488    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5cf239f-c014-4bfe-9479-88a51dbccff3-data\" (UniqueName: \"kubernetes.io/empty-dir/d5cf239f-c014-4bfe-9479-88a51dbccff3-data\") pod \"d5cf239f-c014-4bfe-9479-88a51dbccff3\" (UID: \"d5cf239f-c014-4bfe-9479-88a51dbccff3\") "
Nov 03 20:32:42.034619 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.034523    5591 scope.go:122] "RemoveContainer" containerID="741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3"
Nov 03 20:32:42.034687 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.034616    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5cf239f-c014-4bfe-9479-88a51dbccff3-kube-api-access-rzl9q\" (UniqueName: \"kubernetes.io/projected/d5cf239f-c014-4bfe-9479-88a51dbccff3-kube-api-access-rzl9q\") pod \"d5cf239f-c014-4bfe-9479-88a51dbccff3\" (UID: \"d5cf239f-c014-4bfe-9479-88a51dbccff3\") "
Nov 03 20:32:42.034958 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:42.034930    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3\": container with ID starting with 741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3 not found: ID does not exist" containerID="741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3"
Nov 03 20:32:42.034958 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.034968    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3"} err="failed to get container status \"741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3\": rpc error: code = NotFound desc = could not find container \"741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3\": container with ID starting with 741ba39232ea9b49e228602037f04298cfc1c990250ba9f48733175a9c35a3e3 not found: ID does not exist"
Nov 03 20:32:42.035667 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.035127    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24"]
Nov 03 20:32:42.038231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.038012    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5cf239f-c014-4bfe-9479-88a51dbccff3-kube-api-access-rzl9q" pod "d5cf239f-c014-4bfe-9479-88a51dbccff3" (UID: "d5cf239f-c014-4bfe-9479-88a51dbccff3"). InnerVolumeSpecName "kube-api-access-rzl9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:32:42.044632 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.044602    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 20:32:42.046818 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.046795    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c57db574-5845n"]
Nov 03 20:32:42.054930 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.054902    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"]
Nov 03 20:32:42.088259 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.088222    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5cf239f-c014-4bfe-9479-88a51dbccff3-data" pod "d5cf239f-c014-4bfe-9479-88a51dbccff3" (UID: "d5cf239f-c014-4bfe-9479-88a51dbccff3"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:32:42.101388 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.099999    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 20:32:42.101388 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.100169    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 20:32:42.101388 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.100303    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vf7n8\""
Nov 03 20:32:42.101388 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.100367    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 20:32:42.136875 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.136844    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dac5ea76-74c5-41d8-82bc-8285bc87e535-image-registry-private-configuration\") pod \"image-registry-6c57db574-5845n\" (UID: \"dac5ea76-74c5-41d8-82bc-8285bc87e535\") " pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:32:42.137124 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.136908    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lcqfr\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"
Nov 03 20:32:42.137124 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.137041    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dac5ea76-74c5-41d8-82bc-8285bc87e535-installation-pull-secrets\") pod \"image-registry-6c57db574-5845n\" (UID: \"dac5ea76-74c5-41d8-82bc-8285bc87e535\") " pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:32:42.137124 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.137097    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lcqfr\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"
Nov 03 20:32:42.137311 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.137136    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lcqfr\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"
Nov 03 20:32:42.137311 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.137186    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dac5ea76-74c5-41d8-82bc-8285bc87e535-bound-sa-token\") pod \"image-registry-6c57db574-5845n\" (UID: \"dac5ea76-74c5-41d8-82bc-8285bc87e535\") " pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:32:42.137311 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.137256    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dac5ea76-74c5-41d8-82bc-8285bc87e535-registry-certificates\") pod \"image-registry-6c57db574-5845n\" (UID: \"dac5ea76-74c5-41d8-82bc-8285bc87e535\") " pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:32:42.137311 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.137283    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87xwt\" (UniqueName: \"kubernetes.io/projected/b83ce20e-9504-4cd1-864a-2bbafc41b804-kube-api-access-87xwt\") pod \"maas-api-revoke-sub-403cf98d-5vt58\" (UID: \"b83ce20e-9504-4cd1-864a-2bbafc41b804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58"
Nov 03 20:32:42.137311 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.137311    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfrvn\" (UniqueName: \"kubernetes.io/projected/916d26e6-67e4-49d7-9df8-18c7087aaac1-kube-api-access-kfrvn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lcqfr\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"
Nov 03 20:32:42.137510 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.137403    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dac5ea76-74c5-41d8-82bc-8285bc87e535-ca-trust-extracted\") pod \"image-registry-6c57db574-5845n\" (UID: \"dac5ea76-74c5-41d8-82bc-8285bc87e535\") " pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:32:42.137510 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.137434    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lcqfr\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"
Nov 03 20:32:42.137510 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.137458    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9wl2\" (UniqueName: \"kubernetes.io/projected/3b1b813f-ea90-4f67-9d65-087fc67397f6-kube-api-access-x9wl2\") pod \"maas-api-revoke-sub-fb827799-kpf24\" (UID: \"3b1b813f-ea90-4f67-9d65-087fc67397f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24"
Nov 03 20:32:42.137676 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.137524    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b83ce20e-9504-4cd1-864a-2bbafc41b804-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403cf98d-5vt58\" (UID: \"b83ce20e-9504-4cd1-864a-2bbafc41b804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58"
Nov 03 20:32:42.137676 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.137590    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dac5ea76-74c5-41d8-82bc-8285bc87e535-registry-tls\") pod \"image-registry-6c57db574-5845n\" (UID: \"dac5ea76-74c5-41d8-82bc-8285bc87e535\") " pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:32:42.137676 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.137664    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dac5ea76-74c5-41d8-82bc-8285bc87e535-trusted-ca\") pod \"image-registry-6c57db574-5845n\" (UID: \"dac5ea76-74c5-41d8-82bc-8285bc87e535\") " pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:32:42.137839 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.137725    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/916d26e6-67e4-49d7-9df8-18c7087aaac1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lcqfr\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"
Nov 03 20:32:42.137839 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.137785    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b1b813f-ea90-4f67-9d65-087fc67397f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb827799-kpf24\" (UID: \"3b1b813f-ea90-4f67-9d65-087fc67397f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24"
Nov 03 20:32:42.137839 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.137818    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/916d26e6-67e4-49d7-9df8-18c7087aaac1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lcqfr\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"
Nov 03 20:32:42.137948 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.137876    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b1b813f-ea90-4f67-9d65-087fc67397f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb827799-kpf24\" (UID: \"3b1b813f-ea90-4f67-9d65-087fc67397f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24"
Nov 03 20:32:42.137988 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.137946    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vt9h\" (UniqueName: \"kubernetes.io/projected/dac5ea76-74c5-41d8-82bc-8285bc87e535-kube-api-access-9vt9h\") pod \"image-registry-6c57db574-5845n\" (UID: \"dac5ea76-74c5-41d8-82bc-8285bc87e535\") " pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:32:42.137988 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.137976    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b83ce20e-9504-4cd1-864a-2bbafc41b804-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403cf98d-5vt58\" (UID: \"b83ce20e-9504-4cd1-864a-2bbafc41b804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58"
Nov 03 20:32:42.138114 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.138102    5591 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d5cf239f-c014-4bfe-9479-88a51dbccff3-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:42.138159 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.138118    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzl9q\" (UniqueName: \"kubernetes.io/projected/d5cf239f-c014-4bfe-9479-88a51dbccff3-kube-api-access-rzl9q\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:32:42.239250 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.239104    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/916d26e6-67e4-49d7-9df8-18c7087aaac1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lcqfr\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"
Nov 03 20:32:42.239250 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.239182    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b1b813f-ea90-4f67-9d65-087fc67397f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb827799-kpf24\" (UID: \"3b1b813f-ea90-4f67-9d65-087fc67397f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24"
Nov 03 20:32:42.239250 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.239235    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/916d26e6-67e4-49d7-9df8-18c7087aaac1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lcqfr\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"
Nov 03 20:32:42.239556 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.239276    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b1b813f-ea90-4f67-9d65-087fc67397f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb827799-kpf24\" (UID: \"3b1b813f-ea90-4f67-9d65-087fc67397f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24"
Nov 03 20:32:42.239556 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.239331    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vt9h\" (UniqueName: \"kubernetes.io/projected/dac5ea76-74c5-41d8-82bc-8285bc87e535-kube-api-access-9vt9h\") pod \"image-registry-6c57db574-5845n\" (UID: \"dac5ea76-74c5-41d8-82bc-8285bc87e535\") " pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:32:42.239556 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.239371    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b83ce20e-9504-4cd1-864a-2bbafc41b804-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403cf98d-5vt58\" (UID: \"b83ce20e-9504-4cd1-864a-2bbafc41b804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58"
Nov 03 20:32:42.239556 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.239440    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dac5ea76-74c5-41d8-82bc-8285bc87e535-image-registry-private-configuration\") pod \"image-registry-6c57db574-5845n\" (UID: \"dac5ea76-74c5-41d8-82bc-8285bc87e535\") " pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:32:42.239556 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.239511    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lcqfr\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"
Nov 03 20:32:42.239835 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.239564    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dac5ea76-74c5-41d8-82bc-8285bc87e535-installation-pull-secrets\") pod \"image-registry-6c57db574-5845n\" (UID: \"dac5ea76-74c5-41d8-82bc-8285bc87e535\") " pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:32:42.239835 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.239609    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lcqfr\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"
Nov 03 20:32:42.239835 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.239684    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lcqfr\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"
Nov 03 20:32:42.239835 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.239722    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dac5ea76-74c5-41d8-82bc-8285bc87e535-bound-sa-token\") pod \"image-registry-6c57db574-5845n\" (UID: \"dac5ea76-74c5-41d8-82bc-8285bc87e535\") " pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:32:42.239835 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.239782    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dac5ea76-74c5-41d8-82bc-8285bc87e535-registry-certificates\") pod \"image-registry-6c57db574-5845n\" (UID: \"dac5ea76-74c5-41d8-82bc-8285bc87e535\") " pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:32:42.239835 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.239830    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87xwt\" (UniqueName: \"kubernetes.io/projected/b83ce20e-9504-4cd1-864a-2bbafc41b804-kube-api-access-87xwt\") pod \"maas-api-revoke-sub-403cf98d-5vt58\" (UID: \"b83ce20e-9504-4cd1-864a-2bbafc41b804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58"
Nov 03 20:32:42.240163 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.239872    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfrvn\" (UniqueName: \"kubernetes.io/projected/916d26e6-67e4-49d7-9df8-18c7087aaac1-kube-api-access-kfrvn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lcqfr\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"
Nov 03 20:32:42.240163 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.239922    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dac5ea76-74c5-41d8-82bc-8285bc87e535-ca-trust-extracted\") pod \"image-registry-6c57db574-5845n\" (UID: \"dac5ea76-74c5-41d8-82bc-8285bc87e535\") " pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:32:42.240163 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.239956    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lcqfr\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"
Nov 03 20:32:42.240163 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.239992    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b1b813f-ea90-4f67-9d65-087fc67397f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb827799-kpf24\" (UID: \"3b1b813f-ea90-4f67-9d65-087fc67397f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24"
Nov 03 20:32:42.240163 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.239996    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9wl2\" (UniqueName: \"kubernetes.io/projected/3b1b813f-ea90-4f67-9d65-087fc67397f6-kube-api-access-x9wl2\") pod \"maas-api-revoke-sub-fb827799-kpf24\" (UID: \"3b1b813f-ea90-4f67-9d65-087fc67397f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24"
Nov 03 20:32:42.240476 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.240297    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b83ce20e-9504-4cd1-864a-2bbafc41b804-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403cf98d-5vt58\" (UID: \"b83ce20e-9504-4cd1-864a-2bbafc41b804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58"
Nov 03 20:32:42.240476 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.240368    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dac5ea76-74c5-41d8-82bc-8285bc87e535-registry-tls\") pod \"image-registry-6c57db574-5845n\" (UID: \"dac5ea76-74c5-41d8-82bc-8285bc87e535\") " pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:32:42.240476 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.240456    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dac5ea76-74c5-41d8-82bc-8285bc87e535-trusted-ca\") pod \"image-registry-6c57db574-5845n\" (UID: \"dac5ea76-74c5-41d8-82bc-8285bc87e535\") " pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:32:42.241507 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.241479    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dac5ea76-74c5-41d8-82bc-8285bc87e535-trusted-ca\") pod \"image-registry-6c57db574-5845n\" (UID: \"dac5ea76-74c5-41d8-82bc-8285bc87e535\") " pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:32:42.242594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.242381    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b83ce20e-9504-4cd1-864a-2bbafc41b804-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-403cf98d-5vt58\" (UID: \"b83ce20e-9504-4cd1-864a-2bbafc41b804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58"
Nov 03 20:32:42.242594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.242529    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b83ce20e-9504-4cd1-864a-2bbafc41b804-maas-api-xks-ca\") pod \"maas-api-revoke-sub-403cf98d-5vt58\" (UID: \"b83ce20e-9504-4cd1-864a-2bbafc41b804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58"
Nov 03 20:32:42.242771 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.242647    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/916d26e6-67e4-49d7-9df8-18c7087aaac1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lcqfr\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"
Nov 03 20:32:42.242866 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.242837    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/dac5ea76-74c5-41d8-82bc-8285bc87e535-installation-pull-secrets\") pod \"image-registry-6c57db574-5845n\" (UID: \"dac5ea76-74c5-41d8-82bc-8285bc87e535\") " pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:32:42.242866 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.242855    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b1b813f-ea90-4f67-9d65-087fc67397f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb827799-kpf24\" (UID: \"3b1b813f-ea90-4f67-9d65-087fc67397f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24"
Nov 03 20:32:42.243445 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.243182    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/dac5ea76-74c5-41d8-82bc-8285bc87e535-ca-trust-extracted\") pod \"image-registry-6c57db574-5845n\" (UID: \"dac5ea76-74c5-41d8-82bc-8285bc87e535\") " pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:32:42.243445 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.243401    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lcqfr\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"
Nov 03 20:32:42.243806 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.243699    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/dac5ea76-74c5-41d8-82bc-8285bc87e535-registry-certificates\") pod \"image-registry-6c57db574-5845n\" (UID: \"dac5ea76-74c5-41d8-82bc-8285bc87e535\") " pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:32:42.244079 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.244053    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/dac5ea76-74c5-41d8-82bc-8285bc87e535-image-registry-private-configuration\") pod \"image-registry-6c57db574-5845n\" (UID: \"dac5ea76-74c5-41d8-82bc-8285bc87e535\") " pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:32:42.244415 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.244394    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/916d26e6-67e4-49d7-9df8-18c7087aaac1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lcqfr\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"
Nov 03 20:32:42.244622 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.244599    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lcqfr\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"
Nov 03 20:32:42.245019 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.245000    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/dac5ea76-74c5-41d8-82bc-8285bc87e535-registry-tls\") pod \"image-registry-6c57db574-5845n\" (UID: \"dac5ea76-74c5-41d8-82bc-8285bc87e535\") " pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:32:42.245257 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.245240    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lcqfr\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"
Nov 03 20:32:42.251885 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.251861    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9wl2\" (UniqueName: \"kubernetes.io/projected/3b1b813f-ea90-4f67-9d65-087fc67397f6-kube-api-access-x9wl2\") pod \"maas-api-revoke-sub-fb827799-kpf24\" (UID: \"3b1b813f-ea90-4f67-9d65-087fc67397f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24"
Nov 03 20:32:42.251970 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.251915    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vt9h\" (UniqueName: \"kubernetes.io/projected/dac5ea76-74c5-41d8-82bc-8285bc87e535-kube-api-access-9vt9h\") pod \"image-registry-6c57db574-5845n\" (UID: \"dac5ea76-74c5-41d8-82bc-8285bc87e535\") " pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:32:42.255288 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.255172    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfrvn\" (UniqueName: \"kubernetes.io/projected/916d26e6-67e4-49d7-9df8-18c7087aaac1-kube-api-access-kfrvn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lcqfr\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"
Nov 03 20:32:42.255940 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.255913    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87xwt\" (UniqueName: \"kubernetes.io/projected/b83ce20e-9504-4cd1-864a-2bbafc41b804-kube-api-access-87xwt\") pod \"maas-api-revoke-sub-403cf98d-5vt58\" (UID: \"b83ce20e-9504-4cd1-864a-2bbafc41b804\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58"
Nov 03 20:32:42.256072 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.256051    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dac5ea76-74c5-41d8-82bc-8285bc87e535-bound-sa-token\") pod \"image-registry-6c57db574-5845n\" (UID: \"dac5ea76-74c5-41d8-82bc-8285bc87e535\") " pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:32:42.256116 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.256061    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lcqfr\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"
Nov 03 20:32:42.321637 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.321603    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58"
Nov 03 20:32:42.333623 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.333593    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr"]
Nov 03 20:32:42.342308 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.342282    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-4jqgr"]
Nov 03 20:32:42.345784 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.345755    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24"
Nov 03 20:32:42.401970 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.401740    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"
Nov 03 20:32:42.407682 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.407655    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:32:42.631805 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.630987    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58"]
Nov 03 20:32:42.644590 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:32:42.644506    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb83ce20e_9504_4cd1_864a_2bbafc41b804.slice/crio-73ceff9d76737062ea79a3ce488c18d42f22891e96a829d0c4013952adc87a01 WatchSource:0}: Error finding container 73ceff9d76737062ea79a3ce488c18d42f22891e96a829d0c4013952adc87a01: Status 404 returned error can't find the container with id 73ceff9d76737062ea79a3ce488c18d42f22891e96a829d0c4013952adc87a01
Nov 03 20:32:42.735272 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.725498    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24"]
Nov 03 20:32:42.952530 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.952501    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"]
Nov 03 20:32:42.970740 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.970711    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c57db574-5845n"]
Nov 03 20:32:42.982350 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:32:42.982322    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916d26e6_67e4_49d7_9df8_18c7087aaac1.slice/crio-83a0cfa24a5287bb6b017044b716a35860dcfa55b706c42385861ffd4ca7c6b3 WatchSource:0}: Error finding container 83a0cfa24a5287bb6b017044b716a35860dcfa55b706c42385861ffd4ca7c6b3: Status 404 returned error can't find the container with id 83a0cfa24a5287bb6b017044b716a35860dcfa55b706c42385861ffd4ca7c6b3
Nov 03 20:32:42.983559 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:32:42.983536    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac5ea76_74c5_41d8_82bc_8285bc87e535.slice/crio-71b95222d478a37ae4d783f24f5ff486a52859ff3a32ffd7901ba1f05e47b508 WatchSource:0}: Error finding container 71b95222d478a37ae4d783f24f5ff486a52859ff3a32ffd7901ba1f05e47b508: Status 404 returned error can't find the container with id 71b95222d478a37ae4d783f24f5ff486a52859ff3a32ffd7901ba1f05e47b508
Nov 03 20:32:42.985963 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.985933    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:32:42.986045 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:42.985995    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:32:43.014443 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:43.014410    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr" event={"ID":"916d26e6-67e4-49d7-9df8-18c7087aaac1","Type":"ContainerStarted","Data":"83a0cfa24a5287bb6b017044b716a35860dcfa55b706c42385861ffd4ca7c6b3"}
Nov 03 20:32:43.016100 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:43.016076    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58" event={"ID":"b83ce20e-9504-4cd1-864a-2bbafc41b804","Type":"ContainerStarted","Data":"73ceff9d76737062ea79a3ce488c18d42f22891e96a829d0c4013952adc87a01"}
Nov 03 20:32:43.017758 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:43.017728    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c57db574-5845n" event={"ID":"dac5ea76-74c5-41d8-82bc-8285bc87e535","Type":"ContainerStarted","Data":"71b95222d478a37ae4d783f24f5ff486a52859ff3a32ffd7901ba1f05e47b508"}
Nov 03 20:32:43.020152 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:43.020129    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24" event={"ID":"3b1b813f-ea90-4f67-9d65-087fc67397f6","Type":"ContainerStarted","Data":"a1b60fca0cf06b48f3beebca26da061d377c831ad2145a51cb387e93e86b93d7"}
Nov 03 20:32:43.366483 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:43.366381    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5cf239f-c014-4bfe-9479-88a51dbccff3" path="/var/lib/kubelet/pods/d5cf239f-c014-4bfe-9479-88a51dbccff3/volumes"
Nov 03 20:32:44.033198 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:44.033156    5591 generic.go:356] "Generic (PLEG): container finished" podID="3b1b813f-ea90-4f67-9d65-087fc67397f6" containerID="6e0fa78f1e8d05256ed8dbb4bdd495458816f8fdf8076de81943d33efa1580c9" exitCode=7
Nov 03 20:32:44.033961 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:44.033233    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24" event={"ID":"3b1b813f-ea90-4f67-9d65-087fc67397f6","Type":"ContainerDied","Data":"6e0fa78f1e8d05256ed8dbb4bdd495458816f8fdf8076de81943d33efa1580c9"}
Nov 03 20:32:44.034091 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:44.033661    5591 scope.go:122] "RemoveContainer" containerID="6e0fa78f1e8d05256ed8dbb4bdd495458816f8fdf8076de81943d33efa1580c9"
Nov 03 20:32:44.038832 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:44.038807    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr" event={"ID":"916d26e6-67e4-49d7-9df8-18c7087aaac1","Type":"ContainerStarted","Data":"ac9a2aaa57f42c88a64a76c82f813b610a1dee7deddbff3cdae84db95c6794f9"}
Nov 03 20:32:44.040068 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:44.040044    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"
Nov 03 20:32:44.041398 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:44.041372    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58" event={"ID":"b83ce20e-9504-4cd1-864a-2bbafc41b804","Type":"ContainerStarted","Data":"1407774c9c22914ff40518d05e0f52370da6e3b434ee5f09012e0ae19aa9a20b"}
Nov 03 20:32:44.042676 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:44.042631    5591 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-lcqfr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:32:44.042795 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:44.042700    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr" podUID="916d26e6-67e4-49d7-9df8-18c7087aaac1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:32:44.051453 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:44.051419    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c57db574-5845n" event={"ID":"dac5ea76-74c5-41d8-82bc-8285bc87e535","Type":"ContainerStarted","Data":"c260c7ca6895bef81704cf5146c954d24e5ff31ce284aaaffc5e9dc0924eaee6"}
Nov 03 20:32:44.051583 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:44.051503    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:32:44.088286 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:44.080473    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr" podStartSLOduration=42.08045351 podStartE2EDuration="42.08045351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:44.075772733 +0000 UTC m=+3312.848065782" watchObservedRunningTime="2026-11-03 20:32:44.08045351 +0000 UTC m=+3312.852746529"
Nov 03 20:32:44.106776 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:44.106713    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58" podStartSLOduration=3.1066908460000002 podStartE2EDuration="3.106690846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:44.097477967 +0000 UTC m=+3312.869770986" watchObservedRunningTime="2026-11-03 20:32:44.106690846 +0000 UTC m=+3312.878983866"
Nov 03 20:32:44.127496 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:44.127434    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6c57db574-5845n" podStartSLOduration=42.127413108 podStartE2EDuration="42.127413108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:44.123248172 +0000 UTC m=+3312.895541191" watchObservedRunningTime="2026-11-03 20:32:44.127413108 +0000 UTC m=+3312.899706120"
Nov 03 20:32:45.063130 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:45.063095    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24" event={"ID":"3b1b813f-ea90-4f67-9d65-087fc67397f6","Type":"ContainerStarted","Data":"8326e39c8db96e05152ffde99b75262cba6824b9749c559516a2f9a40f46449c"}
Nov 03 20:32:45.065002 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:45.064970    5591 generic.go:356] "Generic (PLEG): container finished" podID="b83ce20e-9504-4cd1-864a-2bbafc41b804" containerID="1407774c9c22914ff40518d05e0f52370da6e3b434ee5f09012e0ae19aa9a20b" exitCode=22
Nov 03 20:32:45.065154 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:45.065047    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58" event={"ID":"b83ce20e-9504-4cd1-864a-2bbafc41b804","Type":"ContainerDied","Data":"1407774c9c22914ff40518d05e0f52370da6e3b434ee5f09012e0ae19aa9a20b"}
Nov 03 20:32:45.065348 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:45.065330    5591 scope.go:122] "RemoveContainer" containerID="1407774c9c22914ff40518d05e0f52370da6e3b434ee5f09012e0ae19aa9a20b"
Nov 03 20:32:45.065903 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:45.065884    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"
Nov 03 20:32:45.085274 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:45.085228    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24" podStartSLOduration=27.085196603 podStartE2EDuration="27.085196603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:45.083008261 +0000 UTC m=+3313.855301280" watchObservedRunningTime="2026-11-03 20:32:45.085196603 +0000 UTC m=+3313.857489620"
Nov 03 20:32:46.076581 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:46.076534    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58" event={"ID":"b83ce20e-9504-4cd1-864a-2bbafc41b804","Type":"ContainerStarted","Data":"a0eeb0ba535dd9c98daad39af2fdb469be6a9b2ff200049abf73573a92c7c6a1"}
Nov 03 20:32:46.079483 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:46.079448    5591 generic.go:356] "Generic (PLEG): container finished" podID="3b1b813f-ea90-4f67-9d65-087fc67397f6" containerID="8326e39c8db96e05152ffde99b75262cba6824b9749c559516a2f9a40f46449c" exitCode=7
Nov 03 20:32:46.079663 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:46.079532    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24" event={"ID":"3b1b813f-ea90-4f67-9d65-087fc67397f6","Type":"ContainerDied","Data":"8326e39c8db96e05152ffde99b75262cba6824b9749c559516a2f9a40f46449c"}
Nov 03 20:32:46.079663 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:46.079610    5591 scope.go:122] "RemoveContainer" containerID="6e0fa78f1e8d05256ed8dbb4bdd495458816f8fdf8076de81943d33efa1580c9"
Nov 03 20:32:46.079883 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:46.079861    5591 scope.go:122] "RemoveContainer" containerID="8326e39c8db96e05152ffde99b75262cba6824b9749c559516a2f9a40f46449c"
Nov 03 20:32:46.080371 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:46.080344    5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-fb827799-kpf24_odh-ai-gateway-infra(3b1b813f-ea90-4f67-9d65-087fc67397f6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24" podUID="3b1b813f-ea90-4f67-9d65-087fc67397f6"
Nov 03 20:32:47.091770 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:47.091739    5591 scope.go:122] "RemoveContainer" containerID="8326e39c8db96e05152ffde99b75262cba6824b9749c559516a2f9a40f46449c"
Nov 03 20:32:47.092262 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:47.092120    5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-fb827799-kpf24_odh-ai-gateway-infra(3b1b813f-ea90-4f67-9d65-087fc67397f6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24" podUID="3b1b813f-ea90-4f67-9d65-087fc67397f6"
Nov 03 20:32:47.093614 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:47.093591    5591 generic.go:356] "Generic (PLEG): container finished" podID="b83ce20e-9504-4cd1-864a-2bbafc41b804" containerID="a0eeb0ba535dd9c98daad39af2fdb469be6a9b2ff200049abf73573a92c7c6a1" exitCode=22
Nov 03 20:32:47.093744 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:47.093720    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58" event={"ID":"b83ce20e-9504-4cd1-864a-2bbafc41b804","Type":"ContainerDied","Data":"a0eeb0ba535dd9c98daad39af2fdb469be6a9b2ff200049abf73573a92c7c6a1"}
Nov 03 20:32:47.093817 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:47.093789    5591 scope.go:122] "RemoveContainer" containerID="1407774c9c22914ff40518d05e0f52370da6e3b434ee5f09012e0ae19aa9a20b"
Nov 03 20:32:47.093977 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:47.093962    5591 scope.go:122] "RemoveContainer" containerID="a0eeb0ba535dd9c98daad39af2fdb469be6a9b2ff200049abf73573a92c7c6a1"
Nov 03 20:32:47.094360 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:47.094313    5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-403cf98d-5vt58_odh-ai-gateway-infra(b83ce20e-9504-4cd1-864a-2bbafc41b804)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58" podUID="b83ce20e-9504-4cd1-864a-2bbafc41b804"
Nov 03 20:32:48.105100 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:48.105073    5591 scope.go:122] "RemoveContainer" containerID="a0eeb0ba535dd9c98daad39af2fdb469be6a9b2ff200049abf73573a92c7c6a1"
Nov 03 20:32:48.105565 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:48.105446    5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-sub-403cf98d-5vt58_odh-ai-gateway-infra(b83ce20e-9504-4cd1-864a-2bbafc41b804)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58" podUID="b83ce20e-9504-4cd1-864a-2bbafc41b804"
Nov 03 20:32:58.745512 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:32:58.745454    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:32:59.341913 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:32:59.341881    5591 scope.go:122] "RemoveContainer" containerID="8326e39c8db96e05152ffde99b75262cba6824b9749c559516a2f9a40f46449c"
Nov 03 20:33:00.264371 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:00.264332    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24" event={"ID":"3b1b813f-ea90-4f67-9d65-087fc67397f6","Type":"ContainerStarted","Data":"e1dec423e58891146366da2c78616b4160be84183a79be47f761b028bce0e9ba"}
Nov 03 20:33:00.342750 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:00.342721    5591 scope.go:122] "RemoveContainer" containerID="a0eeb0ba535dd9c98daad39af2fdb469be6a9b2ff200049abf73573a92c7c6a1"
Nov 03 20:33:01.274650 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:01.274611    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58" event={"ID":"b83ce20e-9504-4cd1-864a-2bbafc41b804","Type":"ContainerStarted","Data":"f0b481ea971852f34109e0f37697281049edfc5d6e406410b41a9cbb09f0db53"}
Nov 03 20:33:01.277496 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:01.277022    5591 generic.go:356] "Generic (PLEG): container finished" podID="3b1b813f-ea90-4f67-9d65-087fc67397f6" containerID="e1dec423e58891146366da2c78616b4160be84183a79be47f761b028bce0e9ba" exitCode=7
Nov 03 20:33:01.277496 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:01.277094    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24" event={"ID":"3b1b813f-ea90-4f67-9d65-087fc67397f6","Type":"ContainerDied","Data":"e1dec423e58891146366da2c78616b4160be84183a79be47f761b028bce0e9ba"}
Nov 03 20:33:01.277496 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:01.277151    5591 scope.go:122] "RemoveContainer" containerID="8326e39c8db96e05152ffde99b75262cba6824b9749c559516a2f9a40f46449c"
Nov 03 20:33:01.277792 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:01.277625    5591 scope.go:122] "RemoveContainer" containerID="e1dec423e58891146366da2c78616b4160be84183a79be47f761b028bce0e9ba"
Nov 03 20:33:01.278081 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:33:01.278055    5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-sub-fb827799-kpf24_odh-ai-gateway-infra(3b1b813f-ea90-4f67-9d65-087fc67397f6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24" podUID="3b1b813f-ea90-4f67-9d65-087fc67397f6"
Nov 03 20:33:01.374786 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:01.374502    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24"]
Nov 03 20:33:02.321020 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:02.320976    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58"]
Nov 03 20:33:02.341947 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:02.341911    5591 generic.go:356] "Generic (PLEG): container finished" podID="b83ce20e-9504-4cd1-864a-2bbafc41b804" containerID="f0b481ea971852f34109e0f37697281049edfc5d6e406410b41a9cbb09f0db53" exitCode=22
Nov 03 20:33:02.342373 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:02.341970    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58" event={"ID":"b83ce20e-9504-4cd1-864a-2bbafc41b804","Type":"ContainerDied","Data":"f0b481ea971852f34109e0f37697281049edfc5d6e406410b41a9cbb09f0db53"}
Nov 03 20:33:02.342373 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:02.342012    5591 scope.go:122] "RemoveContainer" containerID="a0eeb0ba535dd9c98daad39af2fdb469be6a9b2ff200049abf73573a92c7c6a1"
Nov 03 20:33:02.589375 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:02.589171    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24"
Nov 03 20:33:02.683276 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:02.673303    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58"
Nov 03 20:33:02.735899 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:02.735851    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b1b813f-ea90-4f67-9d65-087fc67397f6-kube-api-access-x9wl2\" (UniqueName: \"kubernetes.io/projected/3b1b813f-ea90-4f67-9d65-087fc67397f6-kube-api-access-x9wl2\") pod \"3b1b813f-ea90-4f67-9d65-087fc67397f6\" (UID: \"3b1b813f-ea90-4f67-9d65-087fc67397f6\") "
Nov 03 20:33:02.736189 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:02.736159    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b1b813f-ea90-4f67-9d65-087fc67397f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b1b813f-ea90-4f67-9d65-087fc67397f6-maas-api-xks-ca\") pod \"3b1b813f-ea90-4f67-9d65-087fc67397f6\" (UID: \"3b1b813f-ea90-4f67-9d65-087fc67397f6\") "
Nov 03 20:33:02.736568 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:02.736545    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b1b813f-ea90-4f67-9d65-087fc67397f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b1b813f-ea90-4f67-9d65-087fc67397f6-maas-api-openshift-ca\") pod \"3b1b813f-ea90-4f67-9d65-087fc67397f6\" (UID: \"3b1b813f-ea90-4f67-9d65-087fc67397f6\") "
Nov 03 20:33:02.737011 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:02.736985    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b1b813f-ea90-4f67-9d65-087fc67397f6-maas-api-openshift-ca" pod "3b1b813f-ea90-4f67-9d65-087fc67397f6" (UID: "3b1b813f-ea90-4f67-9d65-087fc67397f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:02.737908 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:02.737885    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b1b813f-ea90-4f67-9d65-087fc67397f6-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:02.747933 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:02.747899    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b1b813f-ea90-4f67-9d65-087fc67397f6-maas-api-xks-ca" pod "3b1b813f-ea90-4f67-9d65-087fc67397f6" (UID: "3b1b813f-ea90-4f67-9d65-087fc67397f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:02.748392 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:02.748365    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1b813f-ea90-4f67-9d65-087fc67397f6-kube-api-access-x9wl2" pod "3b1b813f-ea90-4f67-9d65-087fc67397f6" (UID: "3b1b813f-ea90-4f67-9d65-087fc67397f6"). InnerVolumeSpecName "kube-api-access-x9wl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:02.839463 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:02.839367    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b83ce20e-9504-4cd1-864a-2bbafc41b804-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b83ce20e-9504-4cd1-864a-2bbafc41b804-maas-api-openshift-ca\") pod \"b83ce20e-9504-4cd1-864a-2bbafc41b804\" (UID: \"b83ce20e-9504-4cd1-864a-2bbafc41b804\") "
Nov 03 20:33:02.839645 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:02.839626    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b83ce20e-9504-4cd1-864a-2bbafc41b804-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b83ce20e-9504-4cd1-864a-2bbafc41b804-maas-api-xks-ca\") pod \"b83ce20e-9504-4cd1-864a-2bbafc41b804\" (UID: \"b83ce20e-9504-4cd1-864a-2bbafc41b804\") "
Nov 03 20:33:02.839708 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:02.839683    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b83ce20e-9504-4cd1-864a-2bbafc41b804-maas-api-openshift-ca" pod "b83ce20e-9504-4cd1-864a-2bbafc41b804" (UID: "b83ce20e-9504-4cd1-864a-2bbafc41b804"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:33:02.839746 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:02.839702    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b83ce20e-9504-4cd1-864a-2bbafc41b804-kube-api-access-87xwt\" (UniqueName: \"kubernetes.io/projected/b83ce20e-9504-4cd1-864a-2bbafc41b804-kube-api-access-87xwt\") pod \"b83ce20e-9504-4cd1-864a-2bbafc41b804\" (UID: \"b83ce20e-9504-4cd1-864a-2bbafc41b804\") "
Nov 03 20:33:02.840611 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:02.840588    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9wl2\" (UniqueName: \"kubernetes.io/projected/3b1b813f-ea90-4f67-9d65-087fc67397f6-kube-api-access-x9wl2\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:02.840708 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:02.840616    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b1b813f-ea90-4f67-9d65-087fc67397f6-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:02.840708 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:02.840632    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b83ce20e-9504-4cd1-864a-2bbafc41b804-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:02.842708 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:02.842676    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b83ce20e-9504-4cd1-864a-2bbafc41b804-maas-api-xks-ca" pod "b83ce20e-9504-4cd1-864a-2bbafc41b804" (UID: "b83ce20e-9504-4cd1-864a-2bbafc41b804"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:33:02.842838 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:02.842736    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b83ce20e-9504-4cd1-864a-2bbafc41b804-kube-api-access-87xwt" pod "b83ce20e-9504-4cd1-864a-2bbafc41b804" (UID: "b83ce20e-9504-4cd1-864a-2bbafc41b804"). InnerVolumeSpecName "kube-api-access-87xwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:33:02.942455 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:02.942407    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b83ce20e-9504-4cd1-864a-2bbafc41b804-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:02.942455 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:02.942448    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87xwt\" (UniqueName: \"kubernetes.io/projected/b83ce20e-9504-4cd1-864a-2bbafc41b804-kube-api-access-87xwt\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:33:03.368632 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:03.368595    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58"
Nov 03 20:33:03.369177 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:03.368760    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58" event={"ID":"b83ce20e-9504-4cd1-864a-2bbafc41b804","Type":"ContainerDied","Data":"73ceff9d76737062ea79a3ce488c18d42f22891e96a829d0c4013952adc87a01"}
Nov 03 20:33:03.369177 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:03.368819    5591 scope.go:122] "RemoveContainer" containerID="f0b481ea971852f34109e0f37697281049edfc5d6e406410b41a9cbb09f0db53"
Nov 03 20:33:03.380796 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:03.380769    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24" event={"ID":"3b1b813f-ea90-4f67-9d65-087fc67397f6","Type":"ContainerDied","Data":"a1b60fca0cf06b48f3beebca26da061d377c831ad2145a51cb387e93e86b93d7"}
Nov 03 20:33:03.380928 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:03.380803    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24"
Nov 03 20:33:03.398655 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:03.398624    5591 scope.go:122] "RemoveContainer" containerID="e1dec423e58891146366da2c78616b4160be84183a79be47f761b028bce0e9ba"
Nov 03 20:33:03.414809 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:03.414775    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24"]
Nov 03 20:33:03.419493 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:03.419469    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb827799-kpf24"]
Nov 03 20:33:03.432858 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:03.432826    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58"]
Nov 03 20:33:03.437458 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:03.437428    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-403cf98d-5vt58"]
Nov 03 20:33:05.069552 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:05.069519    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6c57db574-5845n"
Nov 03 20:33:05.352906 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:05.352823    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b1b813f-ea90-4f67-9d65-087fc67397f6" path="/var/lib/kubelet/pods/3b1b813f-ea90-4f67-9d65-087fc67397f6/volumes"
Nov 03 20:33:05.353493 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:05.353475    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b83ce20e-9504-4cd1-864a-2bbafc41b804" path="/var/lib/kubelet/pods/b83ce20e-9504-4cd1-864a-2bbafc41b804/volumes"
Nov 03 20:33:28.737434 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:33:28.737389    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:33:30.723699 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:33:30.723662    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:33:58.730452 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:33:58.730411    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:34:28.736762 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:34:28.736718    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:34:58.920586 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:34:58.920546    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:35:00.216564 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:00.216523    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:35:04.407937 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:04.407899    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp"]
Nov 03 20:35:09.408494 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:09.408443    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp" podUID="2de58be6-f6e0-4367-ae8a-55cdd99ab480" containerName="payload-pre-processing" containerID="cri-o://5ef82ac9417698e159e614ed07d3fd1c4a554fe6e1cec66b4375c06673a4b30e" gracePeriod=25
Nov 03 20:35:13.065613 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:13.065576    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf"]
Nov 03 20:35:13.066555 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:13.066461    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf" podUID="8c175022-5f51-4cce-8f64-bb399a11ce8d" containerName="maas-api" containerID="cri-o://206495bb49b72f152ad55b326fa84144d13813ece464c80556b5f7262ead93a4" gracePeriod=30
Nov 03 20:35:13.093185 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:13.092962    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf" podUID="8c175022-5f51-4cce-8f64-bb399a11ce8d" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.207:8443/health\": dial tcp 10.134.0.207:8443: connect: connection refused"
Nov 03 20:35:13.650900 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:13.650865    5591 generic.go:356] "Generic (PLEG): container finished" podID="8c175022-5f51-4cce-8f64-bb399a11ce8d" containerID="206495bb49b72f152ad55b326fa84144d13813ece464c80556b5f7262ead93a4" exitCode=0
Nov 03 20:35:13.651523 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:13.651159    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf" event={"ID":"8c175022-5f51-4cce-8f64-bb399a11ce8d","Type":"ContainerDied","Data":"206495bb49b72f152ad55b326fa84144d13813ece464c80556b5f7262ead93a4"}
Nov 03 20:35:13.678814 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:13.678786    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf"
Nov 03 20:35:13.784007 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:13.783970    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c175022-5f51-4cce-8f64-bb399a11ce8d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c175022-5f51-4cce-8f64-bb399a11ce8d-openshift-service-ca\") pod \"8c175022-5f51-4cce-8f64-bb399a11ce8d\" (UID: \"8c175022-5f51-4cce-8f64-bb399a11ce8d\") "
Nov 03 20:35:13.784239 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:13.784133    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c175022-5f51-4cce-8f64-bb399a11ce8d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c175022-5f51-4cce-8f64-bb399a11ce8d-metrics-certs\") pod \"8c175022-5f51-4cce-8f64-bb399a11ce8d\" (UID: \"8c175022-5f51-4cce-8f64-bb399a11ce8d\") "
Nov 03 20:35:13.784239 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:13.784166    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c175022-5f51-4cce-8f64-bb399a11ce8d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c175022-5f51-4cce-8f64-bb399a11ce8d-maas-api-tls\") pod \"8c175022-5f51-4cce-8f64-bb399a11ce8d\" (UID: \"8c175022-5f51-4cce-8f64-bb399a11ce8d\") "
Nov 03 20:35:13.784373 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:13.784290    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c175022-5f51-4cce-8f64-bb399a11ce8d-kube-api-access-655m2\" (UniqueName: \"kubernetes.io/projected/8c175022-5f51-4cce-8f64-bb399a11ce8d-kube-api-access-655m2\") pod \"8c175022-5f51-4cce-8f64-bb399a11ce8d\" (UID: \"8c175022-5f51-4cce-8f64-bb399a11ce8d\") "
Nov 03 20:35:13.784487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:13.784438    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c175022-5f51-4cce-8f64-bb399a11ce8d-openshift-service-ca" pod "8c175022-5f51-4cce-8f64-bb399a11ce8d" (UID: "8c175022-5f51-4cce-8f64-bb399a11ce8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:35:13.785111 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:13.785085    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c175022-5f51-4cce-8f64-bb399a11ce8d-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:13.787114 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:13.787085    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c175022-5f51-4cce-8f64-bb399a11ce8d-kube-api-access-655m2" pod "8c175022-5f51-4cce-8f64-bb399a11ce8d" (UID: "8c175022-5f51-4cce-8f64-bb399a11ce8d"). InnerVolumeSpecName "kube-api-access-655m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:13.787356 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:13.787324    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c175022-5f51-4cce-8f64-bb399a11ce8d-metrics-certs" pod "8c175022-5f51-4cce-8f64-bb399a11ce8d" (UID: "8c175022-5f51-4cce-8f64-bb399a11ce8d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:35:13.787356 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:13.787338    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c175022-5f51-4cce-8f64-bb399a11ce8d-maas-api-tls" pod "8c175022-5f51-4cce-8f64-bb399a11ce8d" (UID: "8c175022-5f51-4cce-8f64-bb399a11ce8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:35:13.886709 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:13.886672    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c175022-5f51-4cce-8f64-bb399a11ce8d-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:13.886709 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:13.886705    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c175022-5f51-4cce-8f64-bb399a11ce8d-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:13.886709 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:13.886718    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-655m2\" (UniqueName: \"kubernetes.io/projected/8c175022-5f51-4cce-8f64-bb399a11ce8d-kube-api-access-655m2\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:14.673294 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:14.673250    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf"
Nov 03 20:35:14.673294 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:14.673265    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf" event={"ID":"8c175022-5f51-4cce-8f64-bb399a11ce8d","Type":"ContainerDied","Data":"d9c20fa8de58d2f56f06131727c59e2a2575ec01b7d52f3ed0d11715c5b757e7"}
Nov 03 20:35:14.673873 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:14.673341    5591 scope.go:122] "RemoveContainer" containerID="206495bb49b72f152ad55b326fa84144d13813ece464c80556b5f7262ead93a4"
Nov 03 20:35:14.711687 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:14.711656    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf"]
Nov 03 20:35:14.719514 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:14.719486    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-b503a8-647fbc7f9f-kl5cf"]
Nov 03 20:35:15.395114 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:15.395064    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c175022-5f51-4cce-8f64-bb399a11ce8d" path="/var/lib/kubelet/pods/8c175022-5f51-4cce-8f64-bb399a11ce8d/volumes"
Nov 03 20:35:15.874391 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:15.874359    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"]
Nov 03 20:35:15.874845 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:15.874577    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6" podUID="9d98020b-67af-47b0-85fc-0190861a7d37" containerName="istio-proxy" containerID="cri-o://3fd856d3df1ad93e35540a9c66f8d6b24e9df7a6bfef4ddabfda520f07793bf2" gracePeriod=30
Nov 03 20:35:18.857415 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:18.857376    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp" podUID="2de58be6-f6e0-4367-ae8a-55cdd99ab480" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:35:21.222904 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.222871    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:35:21.386697 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.386654    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d98020b-67af-47b0-85fc-0190861a7d37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d98020b-67af-47b0-85fc-0190861a7d37-istio-podinfo\") pod \"9d98020b-67af-47b0-85fc-0190861a7d37\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") "
Nov 03 20:35:21.386897 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.386815    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-workload-certs\") pod \"9d98020b-67af-47b0-85fc-0190861a7d37\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") "
Nov 03 20:35:21.386961 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.386940    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d98020b-67af-47b0-85fc-0190861a7d37-kube-api-access-9zgg8\" (UniqueName: \"kubernetes.io/projected/9d98020b-67af-47b0-85fc-0190861a7d37-kube-api-access-9zgg8\") pod \"9d98020b-67af-47b0-85fc-0190861a7d37\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") "
Nov 03 20:35:21.387163 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.387080    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-istio-envoy\") pod \"9d98020b-67af-47b0-85fc-0190861a7d37\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") "
Nov 03 20:35:21.387163 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.387082    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-workload-certs" pod "9d98020b-67af-47b0-85fc-0190861a7d37" (UID: "9d98020b-67af-47b0-85fc-0190861a7d37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:21.387409 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.387171    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d98020b-67af-47b0-85fc-0190861a7d37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d98020b-67af-47b0-85fc-0190861a7d37-istiod-ca-cert\") pod \"9d98020b-67af-47b0-85fc-0190861a7d37\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") "
Nov 03 20:35:21.387409 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.387224    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-workload-socket\") pod \"9d98020b-67af-47b0-85fc-0190861a7d37\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") "
Nov 03 20:35:21.387409 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.387253    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d98020b-67af-47b0-85fc-0190861a7d37-istio-token\" (UniqueName: \"kubernetes.io/projected/9d98020b-67af-47b0-85fc-0190861a7d37-istio-token\") pod \"9d98020b-67af-47b0-85fc-0190861a7d37\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") "
Nov 03 20:35:21.387409 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.387336    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-istio-data\") pod \"9d98020b-67af-47b0-85fc-0190861a7d37\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") "
Nov 03 20:35:21.387409 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.387415    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-credential-socket\") pod \"9d98020b-67af-47b0-85fc-0190861a7d37\" (UID: \"9d98020b-67af-47b0-85fc-0190861a7d37\") "
Nov 03 20:35:21.387799 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.387731    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-workload-socket" pod "9d98020b-67af-47b0-85fc-0190861a7d37" (UID: "9d98020b-67af-47b0-85fc-0190861a7d37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:21.388068 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.387963    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-credential-socket" pod "9d98020b-67af-47b0-85fc-0190861a7d37" (UID: "9d98020b-67af-47b0-85fc-0190861a7d37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:21.388268 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.388148    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-istio-data" pod "9d98020b-67af-47b0-85fc-0190861a7d37" (UID: "9d98020b-67af-47b0-85fc-0190861a7d37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:21.388354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.388266    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d98020b-67af-47b0-85fc-0190861a7d37-istiod-ca-cert" pod "9d98020b-67af-47b0-85fc-0190861a7d37" (UID: "9d98020b-67af-47b0-85fc-0190861a7d37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:35:21.388501 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.388475    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:21.388645 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.388506    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:21.388645 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.388520    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d98020b-67af-47b0-85fc-0190861a7d37-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:21.388645 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.388534    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:21.388645 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.388547    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:21.393948 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.391270    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d98020b-67af-47b0-85fc-0190861a7d37-kube-api-access-9zgg8" pod "9d98020b-67af-47b0-85fc-0190861a7d37" (UID: "9d98020b-67af-47b0-85fc-0190861a7d37"). InnerVolumeSpecName "kube-api-access-9zgg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:21.393948 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.391889    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d98020b-67af-47b0-85fc-0190861a7d37-istio-token" pod "9d98020b-67af-47b0-85fc-0190861a7d37" (UID: "9d98020b-67af-47b0-85fc-0190861a7d37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:21.393948 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.392681    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-istio-envoy" pod "9d98020b-67af-47b0-85fc-0190861a7d37" (UID: "9d98020b-67af-47b0-85fc-0190861a7d37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:35:21.393948 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.393647    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d98020b-67af-47b0-85fc-0190861a7d37-istio-podinfo" pod "9d98020b-67af-47b0-85fc-0190861a7d37" (UID: "9d98020b-67af-47b0-85fc-0190861a7d37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:35:21.490927 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.490841    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d98020b-67af-47b0-85fc-0190861a7d37-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:21.490927 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.490878    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zgg8\" (UniqueName: \"kubernetes.io/projected/9d98020b-67af-47b0-85fc-0190861a7d37-kube-api-access-9zgg8\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:21.490927 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.490894    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d98020b-67af-47b0-85fc-0190861a7d37-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:21.490927 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.490909    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d98020b-67af-47b0-85fc-0190861a7d37-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:21.748065 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.747969    5591 generic.go:356] "Generic (PLEG): container finished" podID="9d98020b-67af-47b0-85fc-0190861a7d37" containerID="3fd856d3df1ad93e35540a9c66f8d6b24e9df7a6bfef4ddabfda520f07793bf2" exitCode=0
Nov 03 20:35:21.748277 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.748104    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"
Nov 03 20:35:21.748277 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.748182    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6" event={"ID":"9d98020b-67af-47b0-85fc-0190861a7d37","Type":"ContainerDied","Data":"3fd856d3df1ad93e35540a9c66f8d6b24e9df7a6bfef4ddabfda520f07793bf2"}
Nov 03 20:35:21.748277 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.748256    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6" event={"ID":"9d98020b-67af-47b0-85fc-0190861a7d37","Type":"ContainerDied","Data":"91901be83901b1893145ebfd625a4dc7cc82a0241ae031987e4f40e930e52ff9"}
Nov 03 20:35:21.748429 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.748286    5591 scope.go:122] "RemoveContainer" containerID="3fd856d3df1ad93e35540a9c66f8d6b24e9df7a6bfef4ddabfda520f07793bf2"
Nov 03 20:35:21.763378 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.763358    5591 scope.go:122] "RemoveContainer" containerID="3fd856d3df1ad93e35540a9c66f8d6b24e9df7a6bfef4ddabfda520f07793bf2"
Nov 03 20:35:21.763666 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:35:21.763646    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fd856d3df1ad93e35540a9c66f8d6b24e9df7a6bfef4ddabfda520f07793bf2\": container with ID starting with 3fd856d3df1ad93e35540a9c66f8d6b24e9df7a6bfef4ddabfda520f07793bf2 not found: ID does not exist" containerID="3fd856d3df1ad93e35540a9c66f8d6b24e9df7a6bfef4ddabfda520f07793bf2"
Nov 03 20:35:21.763743 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.763680    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fd856d3df1ad93e35540a9c66f8d6b24e9df7a6bfef4ddabfda520f07793bf2"} err="failed to get container status \"3fd856d3df1ad93e35540a9c66f8d6b24e9df7a6bfef4ddabfda520f07793bf2\": rpc error: code = NotFound desc = could not find container \"3fd856d3df1ad93e35540a9c66f8d6b24e9df7a6bfef4ddabfda520f07793bf2\": container with ID starting with 3fd856d3df1ad93e35540a9c66f8d6b24e9df7a6bfef4ddabfda520f07793bf2 not found: ID does not exist"
Nov 03 20:35:21.777368 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.777337    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"]
Nov 03 20:35:21.784271 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:21.784227    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-6g2m6"]
Nov 03 20:35:23.353389 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:23.353357    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d98020b-67af-47b0-85fc-0190861a7d37" path="/var/lib/kubelet/pods/9d98020b-67af-47b0-85fc-0190861a7d37/volumes"
Nov 03 20:35:26.129181 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:26.129143    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl"]
Nov 03 20:35:26.134669 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:26.134636    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf"]
Nov 03 20:35:28.742232 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:35:28.738241    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:35:28.888096 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:28.888056    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp" podUID="2de58be6-f6e0-4367-ae8a-55cdd99ab480" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:35:29.771342 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:29.771315    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp"
Nov 03 20:35:29.862865 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:29.862828    5591 generic.go:356] "Generic (PLEG): container finished" podID="2de58be6-f6e0-4367-ae8a-55cdd99ab480" containerID="5ef82ac9417698e159e614ed07d3fd1c4a554fe6e1cec66b4375c06673a4b30e" exitCode=0
Nov 03 20:35:29.863079 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:29.862884    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp"
Nov 03 20:35:29.863079 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:29.862984    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp" event={"ID":"2de58be6-f6e0-4367-ae8a-55cdd99ab480","Type":"ContainerDied","Data":"5ef82ac9417698e159e614ed07d3fd1c4a554fe6e1cec66b4375c06673a4b30e"}
Nov 03 20:35:29.863079 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:29.863048    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp" event={"ID":"2de58be6-f6e0-4367-ae8a-55cdd99ab480","Type":"ContainerDied","Data":"e51bb117ef1ee264f6aa7798f6f11cd1920a3cf84ae5dc6f4027e4b7c8a6378e"}
Nov 03 20:35:29.863326 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:29.863089    5591 scope.go:122] "RemoveContainer" containerID="5ef82ac9417698e159e614ed07d3fd1c4a554fe6e1cec66b4375c06673a4b30e"
Nov 03 20:35:29.876924 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:29.876900    5591 scope.go:122] "RemoveContainer" containerID="5ef82ac9417698e159e614ed07d3fd1c4a554fe6e1cec66b4375c06673a4b30e"
Nov 03 20:35:29.877329 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:35:29.877291    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ef82ac9417698e159e614ed07d3fd1c4a554fe6e1cec66b4375c06673a4b30e\": container with ID starting with 5ef82ac9417698e159e614ed07d3fd1c4a554fe6e1cec66b4375c06673a4b30e not found: ID does not exist" containerID="5ef82ac9417698e159e614ed07d3fd1c4a554fe6e1cec66b4375c06673a4b30e"
Nov 03 20:35:29.877453 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:29.877335    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ef82ac9417698e159e614ed07d3fd1c4a554fe6e1cec66b4375c06673a4b30e"} err="failed to get container status \"5ef82ac9417698e159e614ed07d3fd1c4a554fe6e1cec66b4375c06673a4b30e\": rpc error: code = NotFound desc = could not find container \"5ef82ac9417698e159e614ed07d3fd1c4a554fe6e1cec66b4375c06673a4b30e\": container with ID starting with 5ef82ac9417698e159e614ed07d3fd1c4a554fe6e1cec66b4375c06673a4b30e not found: ID does not exist"
Nov 03 20:35:29.906828 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:29.906799    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2de58be6-f6e0-4367-ae8a-55cdd99ab480-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de58be6-f6e0-4367-ae8a-55cdd99ab480-plugins-config-volume\") pod \"2de58be6-f6e0-4367-ae8a-55cdd99ab480\" (UID: \"2de58be6-f6e0-4367-ae8a-55cdd99ab480\") "
Nov 03 20:35:29.907140 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:29.907120    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2de58be6-f6e0-4367-ae8a-55cdd99ab480-kube-api-access-nfnv7\" (UniqueName: \"kubernetes.io/projected/2de58be6-f6e0-4367-ae8a-55cdd99ab480-kube-api-access-nfnv7\") pod \"2de58be6-f6e0-4367-ae8a-55cdd99ab480\" (UID: \"2de58be6-f6e0-4367-ae8a-55cdd99ab480\") "
Nov 03 20:35:29.907273 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:29.907197    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2de58be6-f6e0-4367-ae8a-55cdd99ab480-plugins-config-volume" pod "2de58be6-f6e0-4367-ae8a-55cdd99ab480" (UID: "2de58be6-f6e0-4367-ae8a-55cdd99ab480"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:35:29.907899 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:29.907879    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2de58be6-f6e0-4367-ae8a-55cdd99ab480-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:29.909424 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:29.909401    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de58be6-f6e0-4367-ae8a-55cdd99ab480-kube-api-access-nfnv7" pod "2de58be6-f6e0-4367-ae8a-55cdd99ab480" (UID: "2de58be6-f6e0-4367-ae8a-55cdd99ab480"). InnerVolumeSpecName "kube-api-access-nfnv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:30.009930 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:30.009841    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfnv7\" (UniqueName: \"kubernetes.io/projected/2de58be6-f6e0-4367-ae8a-55cdd99ab480-kube-api-access-nfnv7\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:30.195309 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:30.195274    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp"]
Nov 03 20:35:30.199651 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:30.199623    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b503a8-78fd8f8ff5-fjfjp"]
Nov 03 20:35:31.131172 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.131125    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl" podUID="c1b5af35-9a4e-4820-ba02-3c5efb129d45" containerName="payload-processing" containerID="cri-o://5f49f4b7a7e1738aca0c1a85f3a78b4481a97ed7c6eeacdd2dc97f3e6f24280a" gracePeriod=25
Nov 03 20:35:31.134910 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.134867    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf" podUID="993ba6c4-13eb-4630-870c-e6018b883996" containerName="payload-pre-processing" containerID="cri-o://76490e2959f68b71475feea24ce66ba9c1134d31b281dca1014e6c087cdc25a0" gracePeriod=25
Nov 03 20:35:31.359780 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.359741    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2de58be6-f6e0-4367-ae8a-55cdd99ab480" path="/var/lib/kubelet/pods/2de58be6-f6e0-4367-ae8a-55cdd99ab480/volumes"
Nov 03 20:35:31.514794 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.514768    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl"
Nov 03 20:35:31.518423 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.518399    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf"
Nov 03 20:35:31.661007 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.660969    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993ba6c4-13eb-4630-870c-e6018b883996-kube-api-access-r9rn7\" (UniqueName: \"kubernetes.io/projected/993ba6c4-13eb-4630-870c-e6018b883996-kube-api-access-r9rn7\") pod \"993ba6c4-13eb-4630-870c-e6018b883996\" (UID: \"993ba6c4-13eb-4630-870c-e6018b883996\") "
Nov 03 20:35:31.661182 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.661157    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1b5af35-9a4e-4820-ba02-3c5efb129d45-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b5af35-9a4e-4820-ba02-3c5efb129d45-plugins-config-volume\") pod \"c1b5af35-9a4e-4820-ba02-3c5efb129d45\" (UID: \"c1b5af35-9a4e-4820-ba02-3c5efb129d45\") "
Nov 03 20:35:31.661301 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.661181    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1b5af35-9a4e-4820-ba02-3c5efb129d45-kube-api-access-7j8zb\" (UniqueName: \"kubernetes.io/projected/c1b5af35-9a4e-4820-ba02-3c5efb129d45-kube-api-access-7j8zb\") pod \"c1b5af35-9a4e-4820-ba02-3c5efb129d45\" (UID: \"c1b5af35-9a4e-4820-ba02-3c5efb129d45\") "
Nov 03 20:35:31.661518 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.661488    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/993ba6c4-13eb-4630-870c-e6018b883996-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993ba6c4-13eb-4630-870c-e6018b883996-plugins-config-volume\") pod \"993ba6c4-13eb-4630-870c-e6018b883996\" (UID: \"993ba6c4-13eb-4630-870c-e6018b883996\") "
Nov 03 20:35:31.661668 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.661543    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b5af35-9a4e-4820-ba02-3c5efb129d45-plugins-config-volume" pod "c1b5af35-9a4e-4820-ba02-3c5efb129d45" (UID: "c1b5af35-9a4e-4820-ba02-3c5efb129d45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:35:31.661830 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.661796    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/993ba6c4-13eb-4630-870c-e6018b883996-plugins-config-volume" pod "993ba6c4-13eb-4630-870c-e6018b883996" (UID: "993ba6c4-13eb-4630-870c-e6018b883996"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:35:31.662249 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.662228    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/993ba6c4-13eb-4630-870c-e6018b883996-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:31.662249 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.662250    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1b5af35-9a4e-4820-ba02-3c5efb129d45-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:31.663331 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.663310    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993ba6c4-13eb-4630-870c-e6018b883996-kube-api-access-r9rn7" pod "993ba6c4-13eb-4630-870c-e6018b883996" (UID: "993ba6c4-13eb-4630-870c-e6018b883996"). InnerVolumeSpecName "kube-api-access-r9rn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:31.663478 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.663461    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b5af35-9a4e-4820-ba02-3c5efb129d45-kube-api-access-7j8zb" pod "c1b5af35-9a4e-4820-ba02-3c5efb129d45" (UID: "c1b5af35-9a4e-4820-ba02-3c5efb129d45"). InnerVolumeSpecName "kube-api-access-7j8zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:31.763692 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.763655    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j8zb\" (UniqueName: \"kubernetes.io/projected/c1b5af35-9a4e-4820-ba02-3c5efb129d45-kube-api-access-7j8zb\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:31.763692 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.763692    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9rn7\" (UniqueName: \"kubernetes.io/projected/993ba6c4-13eb-4630-870c-e6018b883996-kube-api-access-r9rn7\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:31.922457 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.922416    5591 generic.go:356] "Generic (PLEG): container finished" podID="993ba6c4-13eb-4630-870c-e6018b883996" containerID="76490e2959f68b71475feea24ce66ba9c1134d31b281dca1014e6c087cdc25a0" exitCode=0
Nov 03 20:35:31.922676 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.922467    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf"
Nov 03 20:35:31.922676 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.922495    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf" event={"ID":"993ba6c4-13eb-4630-870c-e6018b883996","Type":"ContainerDied","Data":"76490e2959f68b71475feea24ce66ba9c1134d31b281dca1014e6c087cdc25a0"}
Nov 03 20:35:31.922676 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.922581    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf" event={"ID":"993ba6c4-13eb-4630-870c-e6018b883996","Type":"ContainerDied","Data":"529f98f598dd5996432ec5711b9fd662865c244b0d28124c7dac1692e039f045"}
Nov 03 20:35:31.922676 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.922606    5591 scope.go:122] "RemoveContainer" containerID="76490e2959f68b71475feea24ce66ba9c1134d31b281dca1014e6c087cdc25a0"
Nov 03 20:35:31.924627 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.924597    5591 generic.go:356] "Generic (PLEG): container finished" podID="c1b5af35-9a4e-4820-ba02-3c5efb129d45" containerID="5f49f4b7a7e1738aca0c1a85f3a78b4481a97ed7c6eeacdd2dc97f3e6f24280a" exitCode=0
Nov 03 20:35:31.924768 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.924655    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl"
Nov 03 20:35:31.924768 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.924755    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl" event={"ID":"c1b5af35-9a4e-4820-ba02-3c5efb129d45","Type":"ContainerDied","Data":"5f49f4b7a7e1738aca0c1a85f3a78b4481a97ed7c6eeacdd2dc97f3e6f24280a"}
Nov 03 20:35:31.924905 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.924807    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl" event={"ID":"c1b5af35-9a4e-4820-ba02-3c5efb129d45","Type":"ContainerDied","Data":"e42de72e74a8395660316d326b032d2cd5ff34971732a87574cb3dd5271a400f"}
Nov 03 20:35:31.943801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.943765    5591 scope.go:122] "RemoveContainer" containerID="76490e2959f68b71475feea24ce66ba9c1134d31b281dca1014e6c087cdc25a0"
Nov 03 20:35:31.944154 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:35:31.944129    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76490e2959f68b71475feea24ce66ba9c1134d31b281dca1014e6c087cdc25a0\": container with ID starting with 76490e2959f68b71475feea24ce66ba9c1134d31b281dca1014e6c087cdc25a0 not found: ID does not exist" containerID="76490e2959f68b71475feea24ce66ba9c1134d31b281dca1014e6c087cdc25a0"
Nov 03 20:35:31.944318 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.944168    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76490e2959f68b71475feea24ce66ba9c1134d31b281dca1014e6c087cdc25a0"} err="failed to get container status \"76490e2959f68b71475feea24ce66ba9c1134d31b281dca1014e6c087cdc25a0\": rpc error: code = NotFound desc = could not find container \"76490e2959f68b71475feea24ce66ba9c1134d31b281dca1014e6c087cdc25a0\": container with ID starting with 76490e2959f68b71475feea24ce66ba9c1134d31b281dca1014e6c087cdc25a0 not found: ID does not exist"
Nov 03 20:35:31.944318 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.944194    5591 scope.go:122] "RemoveContainer" containerID="5f49f4b7a7e1738aca0c1a85f3a78b4481a97ed7c6eeacdd2dc97f3e6f24280a"
Nov 03 20:35:31.958880 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.958850    5591 scope.go:122] "RemoveContainer" containerID="5f49f4b7a7e1738aca0c1a85f3a78b4481a97ed7c6eeacdd2dc97f3e6f24280a"
Nov 03 20:35:31.959429 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:35:31.959389    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f49f4b7a7e1738aca0c1a85f3a78b4481a97ed7c6eeacdd2dc97f3e6f24280a\": container with ID starting with 5f49f4b7a7e1738aca0c1a85f3a78b4481a97ed7c6eeacdd2dc97f3e6f24280a not found: ID does not exist" containerID="5f49f4b7a7e1738aca0c1a85f3a78b4481a97ed7c6eeacdd2dc97f3e6f24280a"
Nov 03 20:35:31.959640 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.959430    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f49f4b7a7e1738aca0c1a85f3a78b4481a97ed7c6eeacdd2dc97f3e6f24280a"} err="failed to get container status \"5f49f4b7a7e1738aca0c1a85f3a78b4481a97ed7c6eeacdd2dc97f3e6f24280a\": rpc error: code = NotFound desc = could not find container \"5f49f4b7a7e1738aca0c1a85f3a78b4481a97ed7c6eeacdd2dc97f3e6f24280a\": container with ID starting with 5f49f4b7a7e1738aca0c1a85f3a78b4481a97ed7c6eeacdd2dc97f3e6f24280a not found: ID does not exist"
Nov 03 20:35:31.965031 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.964977    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf"]
Nov 03 20:35:31.969888 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.969860    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-942930-78c5bcb55-wmdcf"]
Nov 03 20:35:31.978827 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.978796    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl"]
Nov 03 20:35:31.984050 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:31.984025    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-942930-674dbf8f98-5tmhl"]
Nov 03 20:35:33.165237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:33.163465    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws"]
Nov 03 20:35:33.165237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:33.163724    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws" podUID="1b6e6abf-d08a-4f38-8ad7-6f24074355dd" containerName="maas-api" containerID="cri-o://6688653124a3e4a7eac13bf951c1d39ac6557ee549a90c069bb1a3ec1bded20d" gracePeriod=30
Nov 03 20:35:33.594410 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:33.594353    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="993ba6c4-13eb-4630-870c-e6018b883996" path="/var/lib/kubelet/pods/993ba6c4-13eb-4630-870c-e6018b883996/volumes"
Nov 03 20:35:33.595477 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:33.595453    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1b5af35-9a4e-4820-ba02-3c5efb129d45" path="/var/lib/kubelet/pods/c1b5af35-9a4e-4820-ba02-3c5efb129d45/volumes"
Nov 03 20:35:33.743715 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:33.743686    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws"
Nov 03 20:35:33.792252 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:33.792198    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-metrics-certs\") pod \"1b6e6abf-d08a-4f38-8ad7-6f24074355dd\" (UID: \"1b6e6abf-d08a-4f38-8ad7-6f24074355dd\") "
Nov 03 20:35:33.794027 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:33.792413    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-kube-api-access-5w6lm\" (UniqueName: \"kubernetes.io/projected/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-kube-api-access-5w6lm\") pod \"1b6e6abf-d08a-4f38-8ad7-6f24074355dd\" (UID: \"1b6e6abf-d08a-4f38-8ad7-6f24074355dd\") "
Nov 03 20:35:33.794027 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:33.792451    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-maas-api-tls\") pod \"1b6e6abf-d08a-4f38-8ad7-6f24074355dd\" (UID: \"1b6e6abf-d08a-4f38-8ad7-6f24074355dd\") "
Nov 03 20:35:33.794027 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:33.792512    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-openshift-service-ca\") pod \"1b6e6abf-d08a-4f38-8ad7-6f24074355dd\" (UID: \"1b6e6abf-d08a-4f38-8ad7-6f24074355dd\") "
Nov 03 20:35:33.794027 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:33.793278    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-openshift-service-ca" pod "1b6e6abf-d08a-4f38-8ad7-6f24074355dd" (UID: "1b6e6abf-d08a-4f38-8ad7-6f24074355dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:35:33.798260 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:33.795810    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-maas-api-tls" pod "1b6e6abf-d08a-4f38-8ad7-6f24074355dd" (UID: "1b6e6abf-d08a-4f38-8ad7-6f24074355dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:35:33.798260 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:33.795940    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-kube-api-access-5w6lm" pod "1b6e6abf-d08a-4f38-8ad7-6f24074355dd" (UID: "1b6e6abf-d08a-4f38-8ad7-6f24074355dd"). InnerVolumeSpecName "kube-api-access-5w6lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:33.803650 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:33.803621    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-metrics-certs" pod "1b6e6abf-d08a-4f38-8ad7-6f24074355dd" (UID: "1b6e6abf-d08a-4f38-8ad7-6f24074355dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:35:33.895352 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:33.895257    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:33.895352 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:33.895292    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w6lm\" (UniqueName: \"kubernetes.io/projected/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-kube-api-access-5w6lm\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:33.895352 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:33.895308    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:33.895352 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:33.895326    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6e6abf-d08a-4f38-8ad7-6f24074355dd-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:33.947165 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:33.947128    5591 generic.go:356] "Generic (PLEG): container finished" podID="1b6e6abf-d08a-4f38-8ad7-6f24074355dd" containerID="6688653124a3e4a7eac13bf951c1d39ac6557ee549a90c069bb1a3ec1bded20d" exitCode=0
Nov 03 20:35:33.947387 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:33.947175    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws"
Nov 03 20:35:33.947387 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:33.947228    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws" event={"ID":"1b6e6abf-d08a-4f38-8ad7-6f24074355dd","Type":"ContainerDied","Data":"6688653124a3e4a7eac13bf951c1d39ac6557ee549a90c069bb1a3ec1bded20d"}
Nov 03 20:35:33.947387 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:33.947289    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws" event={"ID":"1b6e6abf-d08a-4f38-8ad7-6f24074355dd","Type":"ContainerDied","Data":"c4bdbf1a2b822cc3232320d1448b69212a718b905ad4b120147ebf64ee104679"}
Nov 03 20:35:33.947387 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:33.947317    5591 scope.go:122] "RemoveContainer" containerID="6688653124a3e4a7eac13bf951c1d39ac6557ee549a90c069bb1a3ec1bded20d"
Nov 03 20:35:33.960587 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:33.960569    5591 scope.go:122] "RemoveContainer" containerID="6688653124a3e4a7eac13bf951c1d39ac6557ee549a90c069bb1a3ec1bded20d"
Nov 03 20:35:33.960892 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:35:33.960871    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6688653124a3e4a7eac13bf951c1d39ac6557ee549a90c069bb1a3ec1bded20d\": container with ID starting with 6688653124a3e4a7eac13bf951c1d39ac6557ee549a90c069bb1a3ec1bded20d not found: ID does not exist" containerID="6688653124a3e4a7eac13bf951c1d39ac6557ee549a90c069bb1a3ec1bded20d"
Nov 03 20:35:33.961005 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:33.960898    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6688653124a3e4a7eac13bf951c1d39ac6557ee549a90c069bb1a3ec1bded20d"} err="failed to get container status \"6688653124a3e4a7eac13bf951c1d39ac6557ee549a90c069bb1a3ec1bded20d\": rpc error: code = NotFound desc = could not find container \"6688653124a3e4a7eac13bf951c1d39ac6557ee549a90c069bb1a3ec1bded20d\": container with ID starting with 6688653124a3e4a7eac13bf951c1d39ac6557ee549a90c069bb1a3ec1bded20d not found: ID does not exist"
Nov 03 20:35:33.973806 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:33.973777    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws"]
Nov 03 20:35:33.979062 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:33.979038    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-942930-98c9b5864-qcqws"]
Nov 03 20:35:35.353176 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:35.353141    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b6e6abf-d08a-4f38-8ad7-6f24074355dd" path="/var/lib/kubelet/pods/1b6e6abf-d08a-4f38-8ad7-6f24074355dd/volumes"
Nov 03 20:35:48.006935 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:48.006903    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4"]
Nov 03 20:35:48.018877 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:48.018841    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp"]
Nov 03 20:35:49.164553 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:49.164510    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:35:53.007489 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:53.007441    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4" podUID="a1cd929e-9b0d-4903-a019-af1f5893c5db" containerName="payload-processing" containerID="cri-o://5e9311bb1019881692967d01d57faf62d326bc64a4cd03edb7d226054c9dd84d" gracePeriod=25
Nov 03 20:35:53.020149 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:53.020109    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp" podUID="1235dbc4-10ee-4c05-b723-49bab294b390" containerName="payload-pre-processing" containerID="cri-o://cb3419aa7659a4d3f99bd58f6adb80f6321ce1acdb4d3faf33fcfcfc2e4d0147" gracePeriod=25
Nov 03 20:35:53.159934 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:53.159901    5591 generic.go:356] "Generic (PLEG): container finished" podID="1235dbc4-10ee-4c05-b723-49bab294b390" containerID="cb3419aa7659a4d3f99bd58f6adb80f6321ce1acdb4d3faf33fcfcfc2e4d0147" exitCode=0
Nov 03 20:35:53.160130 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:53.159959    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp" event={"ID":"1235dbc4-10ee-4c05-b723-49bab294b390","Type":"ContainerDied","Data":"cb3419aa7659a4d3f99bd58f6adb80f6321ce1acdb4d3faf33fcfcfc2e4d0147"}
Nov 03 20:35:53.164972 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:53.164940    5591 generic.go:356] "Generic (PLEG): container finished" podID="a1cd929e-9b0d-4903-a019-af1f5893c5db" containerID="5e9311bb1019881692967d01d57faf62d326bc64a4cd03edb7d226054c9dd84d" exitCode=0
Nov 03 20:35:53.165302 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:53.165006    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4" event={"ID":"a1cd929e-9b0d-4903-a019-af1f5893c5db","Type":"ContainerDied","Data":"5e9311bb1019881692967d01d57faf62d326bc64a4cd03edb7d226054c9dd84d"}
Nov 03 20:35:53.421232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:53.369396    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4"
Nov 03 20:35:53.436971 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:53.436943    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp"
Nov 03 20:35:53.515255 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:53.515168    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1cd929e-9b0d-4903-a019-af1f5893c5db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1cd929e-9b0d-4903-a019-af1f5893c5db-plugins-config-volume\") pod \"a1cd929e-9b0d-4903-a019-af1f5893c5db\" (UID: \"a1cd929e-9b0d-4903-a019-af1f5893c5db\") "
Nov 03 20:35:53.515471 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:53.515445    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1235dbc4-10ee-4c05-b723-49bab294b390-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1235dbc4-10ee-4c05-b723-49bab294b390-plugins-config-volume\") pod \"1235dbc4-10ee-4c05-b723-49bab294b390\" (UID: \"1235dbc4-10ee-4c05-b723-49bab294b390\") "
Nov 03 20:35:53.515631 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:53.515617    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1cd929e-9b0d-4903-a019-af1f5893c5db-kube-api-access-glhsd\" (UniqueName: \"kubernetes.io/projected/a1cd929e-9b0d-4903-a019-af1f5893c5db-kube-api-access-glhsd\") pod \"a1cd929e-9b0d-4903-a019-af1f5893c5db\" (UID: \"a1cd929e-9b0d-4903-a019-af1f5893c5db\") "
Nov 03 20:35:53.515694 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:53.515680    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1235dbc4-10ee-4c05-b723-49bab294b390-kube-api-access-7k8sd\" (UniqueName: \"kubernetes.io/projected/1235dbc4-10ee-4c05-b723-49bab294b390-kube-api-access-7k8sd\") pod \"1235dbc4-10ee-4c05-b723-49bab294b390\" (UID: \"1235dbc4-10ee-4c05-b723-49bab294b390\") "
Nov 03 20:35:53.515927 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:53.515868    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1cd929e-9b0d-4903-a019-af1f5893c5db-plugins-config-volume" pod "a1cd929e-9b0d-4903-a019-af1f5893c5db" (UID: "a1cd929e-9b0d-4903-a019-af1f5893c5db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:35:53.515927 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:53.515899    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1235dbc4-10ee-4c05-b723-49bab294b390-plugins-config-volume" pod "1235dbc4-10ee-4c05-b723-49bab294b390" (UID: "1235dbc4-10ee-4c05-b723-49bab294b390"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:35:53.516576 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:53.516552    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1235dbc4-10ee-4c05-b723-49bab294b390-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:53.516676 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:53.516580    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1cd929e-9b0d-4903-a019-af1f5893c5db-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:53.519555 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:53.519529    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1cd929e-9b0d-4903-a019-af1f5893c5db-kube-api-access-glhsd" pod "a1cd929e-9b0d-4903-a019-af1f5893c5db" (UID: "a1cd929e-9b0d-4903-a019-af1f5893c5db"). InnerVolumeSpecName "kube-api-access-glhsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:53.521514 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:53.521496    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1235dbc4-10ee-4c05-b723-49bab294b390-kube-api-access-7k8sd" pod "1235dbc4-10ee-4c05-b723-49bab294b390" (UID: "1235dbc4-10ee-4c05-b723-49bab294b390"). InnerVolumeSpecName "kube-api-access-7k8sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:35:53.618444 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:53.618352    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glhsd\" (UniqueName: \"kubernetes.io/projected/a1cd929e-9b0d-4903-a019-af1f5893c5db-kube-api-access-glhsd\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:53.618444 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:53.618385    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k8sd\" (UniqueName: \"kubernetes.io/projected/1235dbc4-10ee-4c05-b723-49bab294b390-kube-api-access-7k8sd\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:35:54.181104 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:54.181061    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4"
Nov 03 20:35:54.181680 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:54.181128    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4" event={"ID":"a1cd929e-9b0d-4903-a019-af1f5893c5db","Type":"ContainerDied","Data":"ec41cc1c4fba938f0eff8cc983362f3eafaa04f183c58cbefda6c7db92329378"}
Nov 03 20:35:54.181680 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:54.181283    5591 scope.go:122] "RemoveContainer" containerID="5e9311bb1019881692967d01d57faf62d326bc64a4cd03edb7d226054c9dd84d"
Nov 03 20:35:54.186719 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:54.186684    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp" event={"ID":"1235dbc4-10ee-4c05-b723-49bab294b390","Type":"ContainerDied","Data":"26606330886a296dd390c972b14d0eb59e3e902dd31130769ccd002c03525de2"}
Nov 03 20:35:54.186933 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:54.186844    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp"
Nov 03 20:35:54.197563 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:54.197537    5591 scope.go:122] "RemoveContainer" containerID="cb3419aa7659a4d3f99bd58f6adb80f6321ce1acdb4d3faf33fcfcfc2e4d0147"
Nov 03 20:35:54.221498 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:54.221456    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4"]
Nov 03 20:35:54.225956 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:54.225922    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-88a18e-88bc6576-zdcn4"]
Nov 03 20:35:54.234805 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:54.234777    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp"]
Nov 03 20:35:54.245146 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:54.245117    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-88a18e-7b86865bf5-2nmsp"]
Nov 03 20:35:54.260191 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:35:54.260155    5591 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1235dbc4_10ee_4c05_b723_49bab294b390.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1235dbc4_10ee_4c05_b723_49bab294b390.slice/crio-26606330886a296dd390c972b14d0eb59e3e902dd31130769ccd002c03525de2\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1cd929e_9b0d_4903_a019_af1f5893c5db.slice/crio-ec41cc1c4fba938f0eff8cc983362f3eafaa04f183c58cbefda6c7db92329378\": RecentStats: unable to find data in memory cache]"
Nov 03 20:35:55.353945 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:55.353911    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1235dbc4-10ee-4c05-b723-49bab294b390" path="/var/lib/kubelet/pods/1235dbc4-10ee-4c05-b723-49bab294b390/volumes"
Nov 03 20:35:55.354437 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:35:55.354421    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1cd929e-9b0d-4903-a019-af1f5893c5db" path="/var/lib/kubelet/pods/a1cd929e-9b0d-4903-a019-af1f5893c5db/volumes"
Nov 03 20:35:58.733025 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:35:58.732976    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:36:02.034389 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.034353    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"]
Nov 03 20:36:02.035271 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035254    5591 cpu_manager.go:414] "containerMap: removing container" podUID="b83ce20e-9504-4cd1-864a-2bbafc41b804" containerName="revoke-keys"
Nov 03 20:36:02.035321 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035273    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b83ce20e-9504-4cd1-864a-2bbafc41b804" containerName="revoke-keys"
Nov 03 20:36:02.035321 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035297    5591 cpu_manager.go:414] "containerMap: removing container" podUID="1235dbc4-10ee-4c05-b723-49bab294b390" containerName="payload-pre-processing"
Nov 03 20:36:02.035321 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035303    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1235dbc4-10ee-4c05-b723-49bab294b390" containerName="payload-pre-processing"
Nov 03 20:36:02.035321 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035313    5591 cpu_manager.go:414] "containerMap: removing container" podUID="9d98020b-67af-47b0-85fc-0190861a7d37" containerName="istio-proxy"
Nov 03 20:36:02.035321 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035318    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d98020b-67af-47b0-85fc-0190861a7d37" containerName="istio-proxy"
Nov 03 20:36:02.035490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035327    5591 cpu_manager.go:414] "containerMap: removing container" podUID="3b1b813f-ea90-4f67-9d65-087fc67397f6" containerName="revoke-keys"
Nov 03 20:36:02.035490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035332    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b1b813f-ea90-4f67-9d65-087fc67397f6" containerName="revoke-keys"
Nov 03 20:36:02.035490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035344    5591 cpu_manager.go:414] "containerMap: removing container" podUID="8c175022-5f51-4cce-8f64-bb399a11ce8d" containerName="maas-api"
Nov 03 20:36:02.035490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035350    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c175022-5f51-4cce-8f64-bb399a11ce8d" containerName="maas-api"
Nov 03 20:36:02.035490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035362    5591 cpu_manager.go:414] "containerMap: removing container" podUID="c1b5af35-9a4e-4820-ba02-3c5efb129d45" containerName="payload-processing"
Nov 03 20:36:02.035490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035366    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1b5af35-9a4e-4820-ba02-3c5efb129d45" containerName="payload-processing"
Nov 03 20:36:02.035490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035375    5591 cpu_manager.go:414] "containerMap: removing container" podUID="a1cd929e-9b0d-4903-a019-af1f5893c5db" containerName="payload-processing"
Nov 03 20:36:02.035490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035381    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1cd929e-9b0d-4903-a019-af1f5893c5db" containerName="payload-processing"
Nov 03 20:36:02.035490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035389    5591 cpu_manager.go:414] "containerMap: removing container" podUID="3b1b813f-ea90-4f67-9d65-087fc67397f6" containerName="revoke-keys"
Nov 03 20:36:02.035490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035394    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b1b813f-ea90-4f67-9d65-087fc67397f6" containerName="revoke-keys"
Nov 03 20:36:02.035490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035401    5591 cpu_manager.go:414] "containerMap: removing container" podUID="3b1b813f-ea90-4f67-9d65-087fc67397f6" containerName="revoke-keys"
Nov 03 20:36:02.035490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035405    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b1b813f-ea90-4f67-9d65-087fc67397f6" containerName="revoke-keys"
Nov 03 20:36:02.035490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035418    5591 cpu_manager.go:414] "containerMap: removing container" podUID="2de58be6-f6e0-4367-ae8a-55cdd99ab480" containerName="payload-pre-processing"
Nov 03 20:36:02.035490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035423    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2de58be6-f6e0-4367-ae8a-55cdd99ab480" containerName="payload-pre-processing"
Nov 03 20:36:02.035490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035433    5591 cpu_manager.go:414] "containerMap: removing container" podUID="b83ce20e-9504-4cd1-864a-2bbafc41b804" containerName="revoke-keys"
Nov 03 20:36:02.035490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035437    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b83ce20e-9504-4cd1-864a-2bbafc41b804" containerName="revoke-keys"
Nov 03 20:36:02.035490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035443    5591 cpu_manager.go:414] "containerMap: removing container" podUID="993ba6c4-13eb-4630-870c-e6018b883996" containerName="payload-pre-processing"
Nov 03 20:36:02.035490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035449    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="993ba6c4-13eb-4630-870c-e6018b883996" containerName="payload-pre-processing"
Nov 03 20:36:02.035490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035456    5591 cpu_manager.go:414] "containerMap: removing container" podUID="1b6e6abf-d08a-4f38-8ad7-6f24074355dd" containerName="maas-api"
Nov 03 20:36:02.035490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035461    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b6e6abf-d08a-4f38-8ad7-6f24074355dd" containerName="maas-api"
Nov 03 20:36:02.036284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035581    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b83ce20e-9504-4cd1-864a-2bbafc41b804" containerName="revoke-keys"
Nov 03 20:36:02.036284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035591    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="1235dbc4-10ee-4c05-b723-49bab294b390" containerName="payload-pre-processing"
Nov 03 20:36:02.036284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035596    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d98020b-67af-47b0-85fc-0190861a7d37" containerName="istio-proxy"
Nov 03 20:36:02.036284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035604    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b1b813f-ea90-4f67-9d65-087fc67397f6" containerName="revoke-keys"
Nov 03 20:36:02.036284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035611    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c175022-5f51-4cce-8f64-bb399a11ce8d" containerName="maas-api"
Nov 03 20:36:02.036284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035619    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1b5af35-9a4e-4820-ba02-3c5efb129d45" containerName="payload-processing"
Nov 03 20:36:02.036284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035624    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1cd929e-9b0d-4903-a019-af1f5893c5db" containerName="payload-processing"
Nov 03 20:36:02.036284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035629    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b1b813f-ea90-4f67-9d65-087fc67397f6" containerName="revoke-keys"
Nov 03 20:36:02.036284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035633    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b1b813f-ea90-4f67-9d65-087fc67397f6" containerName="revoke-keys"
Nov 03 20:36:02.036284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035639    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="2de58be6-f6e0-4367-ae8a-55cdd99ab480" containerName="payload-pre-processing"
Nov 03 20:36:02.036284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035645    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b83ce20e-9504-4cd1-864a-2bbafc41b804" containerName="revoke-keys"
Nov 03 20:36:02.036284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035651    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="993ba6c4-13eb-4630-870c-e6018b883996" containerName="payload-pre-processing"
Nov 03 20:36:02.036284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.035655    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b6e6abf-d08a-4f38-8ad7-6f24074355dd" containerName="maas-api"
Nov 03 20:36:02.041354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.041329    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.046328 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.046265    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-bcfde8-openshift-default-dockercfg-6xbwb\""
Nov 03 20:36:02.060831 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.060799    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"]
Nov 03 20:36:02.079913 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.079880    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-istio-envoy\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.080091 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.079942    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e20e21d-9530-4cdb-9927-1376cc371094-istiod-ca-cert\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.080163 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.080084    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e20e21d-9530-4cdb-9927-1376cc371094-istio-podinfo\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.080263 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.080236    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfptp\" (UniqueName: \"kubernetes.io/projected/2e20e21d-9530-4cdb-9927-1376cc371094-kube-api-access-qfptp\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.080320 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.080273    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-workload-certs\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.080361 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.080326    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-credential-socket\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.080404 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.080390    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-istio-data\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.080571 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.080542    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-workload-socket\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.080729 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.080711    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e20e21d-9530-4cdb-9927-1376cc371094-istio-token\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.182809 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.182770    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-istio-envoy\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.183008 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.182824    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e20e21d-9530-4cdb-9927-1376cc371094-istiod-ca-cert\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.183008 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.182863    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e20e21d-9530-4cdb-9927-1376cc371094-istio-podinfo\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.183008 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.182934    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfptp\" (UniqueName: \"kubernetes.io/projected/2e20e21d-9530-4cdb-9927-1376cc371094-kube-api-access-qfptp\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.183008 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.182968    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-workload-certs\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.183252 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.183007    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-credential-socket\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.183252 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.183058    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-istio-data\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.183252 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.183182    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-workload-socket\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.183518 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.183313    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e20e21d-9530-4cdb-9927-1376cc371094-istio-token\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.183518 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.183380    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-workload-certs\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.183713 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.183569    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-credential-socket\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.183713 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.183656    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-istio-data\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.183713 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.183676    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e20e21d-9530-4cdb-9927-1376cc371094-istiod-ca-cert\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.184087 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.184060    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-workload-socket\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.185540 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.185510    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-istio-envoy\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.185744 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.185727    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e20e21d-9530-4cdb-9927-1376cc371094-istio-podinfo\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.195301 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.195277    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e20e21d-9530-4cdb-9927-1376cc371094-istio-token\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.195406 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.195326    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfptp\" (UniqueName: \"kubernetes.io/projected/2e20e21d-9530-4cdb-9927-1376cc371094-kube-api-access-qfptp\") pod \"e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") " pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.379181 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.379092    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:02.682783 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.682750    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"]
Nov 03 20:36:02.683639 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:36:02.683191    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e20e21d_9530_4cdb_9927_1376cc371094.slice/crio-bd895216e5ceb77f3175cd63edaf928db942b99c6edf5ee373e3a4ebcb9762be WatchSource:0}: Error finding container bd895216e5ceb77f3175cd63edaf928db942b99c6edf5ee373e3a4ebcb9762be: Status 404 returned error can't find the container with id bd895216e5ceb77f3175cd63edaf928db942b99c6edf5ee373e3a4ebcb9762be
Nov 03 20:36:02.692279 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.692234    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:36:02.692432 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.692331    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:36:02.692432 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:02.692393    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:36:03.357803 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:03.357761    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk" event={"ID":"2e20e21d-9530-4cdb-9927-1376cc371094","Type":"ContainerStarted","Data":"011d89e379e254ef23e9388d9bd2cd6ed574910cad0153ae7562280c867affb8"}
Nov 03 20:36:03.358296 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:03.357832    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk" event={"ID":"2e20e21d-9530-4cdb-9927-1376cc371094","Type":"ContainerStarted","Data":"bd895216e5ceb77f3175cd63edaf928db942b99c6edf5ee373e3a4ebcb9762be"}
Nov 03 20:36:03.380171 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:03.379881    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:03.382143 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:03.382113    5591 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Nov 03 20:36:03.382313 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:03.382165    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk" podUID="2e20e21d-9530-4cdb-9927-1376cc371094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Nov 03 20:36:03.384644 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:03.384601    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk" podStartSLOduration=1.384585907 podStartE2EDuration="1.384585907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:03.379044124 +0000 UTC m=+3512.151337143" watchObservedRunningTime="2026-11-03 20:36:03.384585907 +0000 UTC m=+3512.156878924"
Nov 03 20:36:04.401318 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:04.388642    5591 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Nov 03 20:36:04.401318 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:04.388723    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk" podUID="2e20e21d-9530-4cdb-9927-1376cc371094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Nov 03 20:36:05.400228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:05.395100    5591 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Nov 03 20:36:05.400228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:05.395163    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk" podUID="2e20e21d-9530-4cdb-9927-1376cc371094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Nov 03 20:36:06.396755 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:06.396704    5591 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Nov 03 20:36:06.397327 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:06.396800    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk" podUID="2e20e21d-9530-4cdb-9927-1376cc371094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Nov 03 20:36:07.400230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:07.396694    5591 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Nov 03 20:36:07.400230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:07.396768    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk" podUID="2e20e21d-9530-4cdb-9927-1376cc371094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Nov 03 20:36:08.334123 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:08.334083    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"]
Nov 03 20:36:08.334852 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:08.334773    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l" podUID="4a55d034-3d5e-41f1-ae14-b741d8b0c2a5" containerName="istio-proxy" containerID="cri-o://3614abd781be46c274bc81b850b2a40344ae5f0eba3a5bfb56dd0d3d2d911d46" gracePeriod=30
Nov 03 20:36:08.401643 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:08.401602    5591 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Nov 03 20:36:08.407273 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:08.402246    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk" podUID="2e20e21d-9530-4cdb-9927-1376cc371094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Nov 03 20:36:09.380175 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:09.380128    5591 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused" start-of-body=
Nov 03 20:36:09.380718 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:09.380195    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk" podUID="2e20e21d-9530-4cdb-9927-1376cc371094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.20:15021/healthz/ready\": dial tcp 10.134.1.20:15021: connect: connection refused"
Nov 03 20:36:10.384739 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:10.384706    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:10.385242 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:10.385222    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:10.385905 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:10.385888    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:36:13.618982 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.618885    5591 generic.go:356] "Generic (PLEG): container finished" podID="4a55d034-3d5e-41f1-ae14-b741d8b0c2a5" containerID="3614abd781be46c274bc81b850b2a40344ae5f0eba3a5bfb56dd0d3d2d911d46" exitCode=0
Nov 03 20:36:13.618982 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.618957    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l" event={"ID":"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5","Type":"ContainerDied","Data":"3614abd781be46c274bc81b850b2a40344ae5f0eba3a5bfb56dd0d3d2d911d46"}
Nov 03 20:36:13.682064 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.682033    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:36:13.789514 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.789481    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-workload-certs\") pod \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") "
Nov 03 20:36:13.789694 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.789534    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-credential-socket\") pod \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") "
Nov 03 20:36:13.789694 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.789574    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-workload-socket\") pod \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") "
Nov 03 20:36:13.789777 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.789761    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istiod-ca-cert\") pod \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") "
Nov 03 20:36:13.789819 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.789770    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-workload-certs" pod "4a55d034-3d5e-41f1-ae14-b741d8b0c2a5" (UID: "4a55d034-3d5e-41f1-ae14-b741d8b0c2a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:13.789857 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.789829    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-token\" (UniqueName: \"kubernetes.io/projected/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-token\") pod \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") "
Nov 03 20:36:13.789918 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.789894    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-credential-socket" pod "4a55d034-3d5e-41f1-ae14-b741d8b0c2a5" (UID: "4a55d034-3d5e-41f1-ae14-b741d8b0c2a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:13.789964 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.789905    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-workload-socket" pod "4a55d034-3d5e-41f1-ae14-b741d8b0c2a5" (UID: "4a55d034-3d5e-41f1-ae14-b741d8b0c2a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:13.789964 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.789874    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-podinfo\") pod \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") "
Nov 03 20:36:13.790073 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.790043    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-kube-api-access-swvdr\" (UniqueName: \"kubernetes.io/projected/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-kube-api-access-swvdr\") pod \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") "
Nov 03 20:36:13.790176 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.790099    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-data\") pod \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") "
Nov 03 20:36:13.790176 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.790154    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-envoy\") pod \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\" (UID: \"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5\") "
Nov 03 20:36:13.790333 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.790150    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istiod-ca-cert" pod "4a55d034-3d5e-41f1-ae14-b741d8b0c2a5" (UID: "4a55d034-3d5e-41f1-ae14-b741d8b0c2a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:36:13.790431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.790407    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-data" pod "4a55d034-3d5e-41f1-ae14-b741d8b0c2a5" (UID: "4a55d034-3d5e-41f1-ae14-b741d8b0c2a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:13.791488 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.791458    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:13.791636 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.791489    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:13.791636 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.791507    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:13.791636 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.791521    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:13.791636 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.791535    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:13.792453 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.792432    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-token" pod "4a55d034-3d5e-41f1-ae14-b741d8b0c2a5" (UID: "4a55d034-3d5e-41f1-ae14-b741d8b0c2a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:13.792721 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.792695    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-kube-api-access-swvdr" pod "4a55d034-3d5e-41f1-ae14-b741d8b0c2a5" (UID: "4a55d034-3d5e-41f1-ae14-b741d8b0c2a5"). InnerVolumeSpecName "kube-api-access-swvdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:13.792820 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.792794    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-podinfo" pod "4a55d034-3d5e-41f1-ae14-b741d8b0c2a5" (UID: "4a55d034-3d5e-41f1-ae14-b741d8b0c2a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:36:13.792820 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.792799    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-envoy" pod "4a55d034-3d5e-41f1-ae14-b741d8b0c2a5" (UID: "4a55d034-3d5e-41f1-ae14-b741d8b0c2a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:13.893594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.893493    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:13.893594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.893524    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:13.893594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.893534    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swvdr\" (UniqueName: \"kubernetes.io/projected/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-kube-api-access-swvdr\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:13.893594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:13.893545    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:14.628709 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:14.628674    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l" event={"ID":"4a55d034-3d5e-41f1-ae14-b741d8b0c2a5","Type":"ContainerDied","Data":"092781921a56ad0b953512dd5fa8ec0b478d505e64bc57cf62f7cdae07a66f98"}
Nov 03 20:36:14.629181 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:14.628732    5591 scope.go:122] "RemoveContainer" containerID="3614abd781be46c274bc81b850b2a40344ae5f0eba3a5bfb56dd0d3d2d911d46"
Nov 03 20:36:14.629181 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:14.628689    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"
Nov 03 20:36:14.708357 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:14.708320    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"]
Nov 03 20:36:14.716641 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:14.716607    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e1546-openshift-default-657f9d4657-p545l"]
Nov 03 20:36:15.352820 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:15.352781    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a55d034-3d5e-41f1-ae14-b741d8b0c2a5" path="/var/lib/kubelet/pods/4a55d034-3d5e-41f1-ae14-b741d8b0c2a5/volumes"
Nov 03 20:36:16.205117 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:16.205079    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg"]
Nov 03 20:36:16.212313 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:16.212281    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz"]
Nov 03 20:36:19.182631 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:19.182592    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8"]
Nov 03 20:36:19.183131 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:19.182909    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8" podUID="0366c67a-080b-4983-8057-b0d9ef562477" containerName="maas-api" containerID="cri-o://006f25552fde813c3298da814e906a6144795fbd26cf455a96c17dc244d0797f" gracePeriod=30
Nov 03 20:36:19.683784 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:19.683751    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8"
Nov 03 20:36:19.700549 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:19.700508    5591 generic.go:356] "Generic (PLEG): container finished" podID="0366c67a-080b-4983-8057-b0d9ef562477" containerID="006f25552fde813c3298da814e906a6144795fbd26cf455a96c17dc244d0797f" exitCode=0
Nov 03 20:36:19.700745 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:19.700591    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8" event={"ID":"0366c67a-080b-4983-8057-b0d9ef562477","Type":"ContainerDied","Data":"006f25552fde813c3298da814e906a6144795fbd26cf455a96c17dc244d0797f"}
Nov 03 20:36:19.700745 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:19.700670    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8" event={"ID":"0366c67a-080b-4983-8057-b0d9ef562477","Type":"ContainerDied","Data":"c9e19c1a3528ce5ba9490f20319a457d2e575ccdf2a1c8be46821f656f6e392e"}
Nov 03 20:36:19.700745 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:19.700598    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8"
Nov 03 20:36:19.700745 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:19.700693    5591 scope.go:122] "RemoveContainer" containerID="006f25552fde813c3298da814e906a6144795fbd26cf455a96c17dc244d0797f"
Nov 03 20:36:19.737020 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:19.736998    5591 scope.go:122] "RemoveContainer" containerID="006f25552fde813c3298da814e906a6144795fbd26cf455a96c17dc244d0797f"
Nov 03 20:36:19.737783 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:36:19.737752    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"006f25552fde813c3298da814e906a6144795fbd26cf455a96c17dc244d0797f\": container with ID starting with 006f25552fde813c3298da814e906a6144795fbd26cf455a96c17dc244d0797f not found: ID does not exist" containerID="006f25552fde813c3298da814e906a6144795fbd26cf455a96c17dc244d0797f"
Nov 03 20:36:19.737919 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:19.737790    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"006f25552fde813c3298da814e906a6144795fbd26cf455a96c17dc244d0797f"} err="failed to get container status \"006f25552fde813c3298da814e906a6144795fbd26cf455a96c17dc244d0797f\": rpc error: code = NotFound desc = could not find container \"006f25552fde813c3298da814e906a6144795fbd26cf455a96c17dc244d0797f\": container with ID starting with 006f25552fde813c3298da814e906a6144795fbd26cf455a96c17dc244d0797f not found: ID does not exist"
Nov 03 20:36:19.743266 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:19.740854    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0366c67a-080b-4983-8057-b0d9ef562477-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0366c67a-080b-4983-8057-b0d9ef562477-openshift-service-ca\") pod \"0366c67a-080b-4983-8057-b0d9ef562477\" (UID: \"0366c67a-080b-4983-8057-b0d9ef562477\") "
Nov 03 20:36:19.743266 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:19.740961    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0366c67a-080b-4983-8057-b0d9ef562477-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0366c67a-080b-4983-8057-b0d9ef562477-metrics-certs\") pod \"0366c67a-080b-4983-8057-b0d9ef562477\" (UID: \"0366c67a-080b-4983-8057-b0d9ef562477\") "
Nov 03 20:36:19.743266 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:19.741088    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0366c67a-080b-4983-8057-b0d9ef562477-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0366c67a-080b-4983-8057-b0d9ef562477-maas-api-tls\") pod \"0366c67a-080b-4983-8057-b0d9ef562477\" (UID: \"0366c67a-080b-4983-8057-b0d9ef562477\") "
Nov 03 20:36:19.743266 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:19.741326    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0366c67a-080b-4983-8057-b0d9ef562477-kube-api-access-h4hdr\" (UniqueName: \"kubernetes.io/projected/0366c67a-080b-4983-8057-b0d9ef562477-kube-api-access-h4hdr\") pod \"0366c67a-080b-4983-8057-b0d9ef562477\" (UID: \"0366c67a-080b-4983-8057-b0d9ef562477\") "
Nov 03 20:36:19.743622 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:19.743600    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0366c67a-080b-4983-8057-b0d9ef562477-openshift-service-ca" pod "0366c67a-080b-4983-8057-b0d9ef562477" (UID: "0366c67a-080b-4983-8057-b0d9ef562477"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:36:19.748575 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:19.748548    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0366c67a-080b-4983-8057-b0d9ef562477-metrics-certs" pod "0366c67a-080b-4983-8057-b0d9ef562477" (UID: "0366c67a-080b-4983-8057-b0d9ef562477"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:19.748744 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:19.748556    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0366c67a-080b-4983-8057-b0d9ef562477-kube-api-access-h4hdr" pod "0366c67a-080b-4983-8057-b0d9ef562477" (UID: "0366c67a-080b-4983-8057-b0d9ef562477"). InnerVolumeSpecName "kube-api-access-h4hdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:19.748872 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:19.748748    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0366c67a-080b-4983-8057-b0d9ef562477-maas-api-tls" pod "0366c67a-080b-4983-8057-b0d9ef562477" (UID: "0366c67a-080b-4983-8057-b0d9ef562477"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:19.864231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:19.850966    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0366c67a-080b-4983-8057-b0d9ef562477-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:19.864231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:19.851004    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0366c67a-080b-4983-8057-b0d9ef562477-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:19.864231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:19.851020    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4hdr\" (UniqueName: \"kubernetes.io/projected/0366c67a-080b-4983-8057-b0d9ef562477-kube-api-access-h4hdr\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:19.864231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:19.851035    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0366c67a-080b-4983-8057-b0d9ef562477-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:20.028471 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:20.028436    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8"]
Nov 03 20:36:20.032528 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:20.032507    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4fb1d0-6cdb8bccc6-tcjk8"]
Nov 03 20:36:21.206022 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.205986    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg" podUID="2339afa1-517b-49cc-9d17-a037371e38e8" containerName="payload-processing" containerID="cri-o://b09ca31bbf40bcec53b3632b26ba4f993271685fe8afd7162539a960a41ba789" gracePeriod=25
Nov 03 20:36:21.212679 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.212645    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz" podUID="75752636-c0d6-46d6-b2d2-da4122853fac" containerName="payload-pre-processing" containerID="cri-o://d13147fa6cf8257ec8edeaea1229c525de295de0cda7bd5f71f508f996687fd3" gracePeriod=25
Nov 03 20:36:21.370064 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.358297    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0366c67a-080b-4983-8057-b0d9ef562477" path="/var/lib/kubelet/pods/0366c67a-080b-4983-8057-b0d9ef562477/volumes"
Nov 03 20:36:21.565653 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.565623    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg"
Nov 03 20:36:21.585686 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.585657    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2339afa1-517b-49cc-9d17-a037371e38e8-kube-api-access-cwtkc\" (UniqueName: \"kubernetes.io/projected/2339afa1-517b-49cc-9d17-a037371e38e8-kube-api-access-cwtkc\") pod \"2339afa1-517b-49cc-9d17-a037371e38e8\" (UID: \"2339afa1-517b-49cc-9d17-a037371e38e8\") "
Nov 03 20:36:21.585870 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.585696    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2339afa1-517b-49cc-9d17-a037371e38e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2339afa1-517b-49cc-9d17-a037371e38e8-plugins-config-volume\") pod \"2339afa1-517b-49cc-9d17-a037371e38e8\" (UID: \"2339afa1-517b-49cc-9d17-a037371e38e8\") "
Nov 03 20:36:21.586601 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.586572    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2339afa1-517b-49cc-9d17-a037371e38e8-plugins-config-volume" pod "2339afa1-517b-49cc-9d17-a037371e38e8" (UID: "2339afa1-517b-49cc-9d17-a037371e38e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:36:21.599454 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.599428    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2339afa1-517b-49cc-9d17-a037371e38e8-kube-api-access-cwtkc" pod "2339afa1-517b-49cc-9d17-a037371e38e8" (UID: "2339afa1-517b-49cc-9d17-a037371e38e8"). InnerVolumeSpecName "kube-api-access-cwtkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:21.619070 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.619048    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz"
Nov 03 20:36:21.687537 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.687498    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75752636-c0d6-46d6-b2d2-da4122853fac-kube-api-access-fpkmn\" (UniqueName: \"kubernetes.io/projected/75752636-c0d6-46d6-b2d2-da4122853fac-kube-api-access-fpkmn\") pod \"75752636-c0d6-46d6-b2d2-da4122853fac\" (UID: \"75752636-c0d6-46d6-b2d2-da4122853fac\") "
Nov 03 20:36:21.687726 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.687608    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75752636-c0d6-46d6-b2d2-da4122853fac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75752636-c0d6-46d6-b2d2-da4122853fac-plugins-config-volume\") pod \"75752636-c0d6-46d6-b2d2-da4122853fac\" (UID: \"75752636-c0d6-46d6-b2d2-da4122853fac\") "
Nov 03 20:36:21.687996 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.687973    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75752636-c0d6-46d6-b2d2-da4122853fac-plugins-config-volume" pod "75752636-c0d6-46d6-b2d2-da4122853fac" (UID: "75752636-c0d6-46d6-b2d2-da4122853fac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:36:21.688535 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.688511    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75752636-c0d6-46d6-b2d2-da4122853fac-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:21.688535 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.688534    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwtkc\" (UniqueName: \"kubernetes.io/projected/2339afa1-517b-49cc-9d17-a037371e38e8-kube-api-access-cwtkc\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:21.688696 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.688544    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2339afa1-517b-49cc-9d17-a037371e38e8-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:21.689958 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.689933    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75752636-c0d6-46d6-b2d2-da4122853fac-kube-api-access-fpkmn" pod "75752636-c0d6-46d6-b2d2-da4122853fac" (UID: "75752636-c0d6-46d6-b2d2-da4122853fac"). InnerVolumeSpecName "kube-api-access-fpkmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:21.723565 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.723515    5591 generic.go:356] "Generic (PLEG): container finished" podID="75752636-c0d6-46d6-b2d2-da4122853fac" containerID="d13147fa6cf8257ec8edeaea1229c525de295de0cda7bd5f71f508f996687fd3" exitCode=0
Nov 03 20:36:21.723768 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.723569    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz"
Nov 03 20:36:21.723768 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.723587    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz" event={"ID":"75752636-c0d6-46d6-b2d2-da4122853fac","Type":"ContainerDied","Data":"d13147fa6cf8257ec8edeaea1229c525de295de0cda7bd5f71f508f996687fd3"}
Nov 03 20:36:21.723768 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.723649    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz" event={"ID":"75752636-c0d6-46d6-b2d2-da4122853fac","Type":"ContainerDied","Data":"8360583ae05bfdcabee0b6307075ddc88c3a976a93ad6fa0922788166bb79a8f"}
Nov 03 20:36:21.723768 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.723684    5591 scope.go:122] "RemoveContainer" containerID="d13147fa6cf8257ec8edeaea1229c525de295de0cda7bd5f71f508f996687fd3"
Nov 03 20:36:21.725978 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.725842    5591 generic.go:356] "Generic (PLEG): container finished" podID="2339afa1-517b-49cc-9d17-a037371e38e8" containerID="b09ca31bbf40bcec53b3632b26ba4f993271685fe8afd7162539a960a41ba789" exitCode=0
Nov 03 20:36:21.725978 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.725907    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg"
Nov 03 20:36:21.726189 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.726011    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg" event={"ID":"2339afa1-517b-49cc-9d17-a037371e38e8","Type":"ContainerDied","Data":"b09ca31bbf40bcec53b3632b26ba4f993271685fe8afd7162539a960a41ba789"}
Nov 03 20:36:21.726189 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.726065    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg" event={"ID":"2339afa1-517b-49cc-9d17-a037371e38e8","Type":"ContainerDied","Data":"0895916030a8f3980de5afd5724ab5cb71044e2f3706852065cc184fee8d0e0c"}
Nov 03 20:36:21.740797 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.740774    5591 scope.go:122] "RemoveContainer" containerID="d13147fa6cf8257ec8edeaea1229c525de295de0cda7bd5f71f508f996687fd3"
Nov 03 20:36:21.741091 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:36:21.741075    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d13147fa6cf8257ec8edeaea1229c525de295de0cda7bd5f71f508f996687fd3\": container with ID starting with d13147fa6cf8257ec8edeaea1229c525de295de0cda7bd5f71f508f996687fd3 not found: ID does not exist" containerID="d13147fa6cf8257ec8edeaea1229c525de295de0cda7bd5f71f508f996687fd3"
Nov 03 20:36:21.741157 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.741098    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13147fa6cf8257ec8edeaea1229c525de295de0cda7bd5f71f508f996687fd3"} err="failed to get container status \"d13147fa6cf8257ec8edeaea1229c525de295de0cda7bd5f71f508f996687fd3\": rpc error: code = NotFound desc = could not find container \"d13147fa6cf8257ec8edeaea1229c525de295de0cda7bd5f71f508f996687fd3\": container with ID starting with d13147fa6cf8257ec8edeaea1229c525de295de0cda7bd5f71f508f996687fd3 not found: ID does not exist"
Nov 03 20:36:21.741157 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.741114    5591 scope.go:122] "RemoveContainer" containerID="b09ca31bbf40bcec53b3632b26ba4f993271685fe8afd7162539a960a41ba789"
Nov 03 20:36:21.752891 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.752872    5591 scope.go:122] "RemoveContainer" containerID="b09ca31bbf40bcec53b3632b26ba4f993271685fe8afd7162539a960a41ba789"
Nov 03 20:36:21.753218 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:36:21.753178    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b09ca31bbf40bcec53b3632b26ba4f993271685fe8afd7162539a960a41ba789\": container with ID starting with b09ca31bbf40bcec53b3632b26ba4f993271685fe8afd7162539a960a41ba789 not found: ID does not exist" containerID="b09ca31bbf40bcec53b3632b26ba4f993271685fe8afd7162539a960a41ba789"
Nov 03 20:36:21.753320 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.753228    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b09ca31bbf40bcec53b3632b26ba4f993271685fe8afd7162539a960a41ba789"} err="failed to get container status \"b09ca31bbf40bcec53b3632b26ba4f993271685fe8afd7162539a960a41ba789\": rpc error: code = NotFound desc = could not find container \"b09ca31bbf40bcec53b3632b26ba4f993271685fe8afd7162539a960a41ba789\": container with ID starting with b09ca31bbf40bcec53b3632b26ba4f993271685fe8afd7162539a960a41ba789 not found: ID does not exist"
Nov 03 20:36:21.755611 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.755512    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz"]
Nov 03 20:36:21.761105 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.761085    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4fb1d0-7958c77cb7-cs7sz"]
Nov 03 20:36:21.774816 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.774751    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg"]
Nov 03 20:36:21.778511 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.778488    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4fb1d0-86df6f5d94-b8pcg"]
Nov 03 20:36:21.789926 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:21.789900    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpkmn\" (UniqueName: \"kubernetes.io/projected/75752636-c0d6-46d6-b2d2-da4122853fac-kube-api-access-fpkmn\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:23.356597 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:23.356552    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2339afa1-517b-49cc-9d17-a037371e38e8" path="/var/lib/kubelet/pods/2339afa1-517b-49cc-9d17-a037371e38e8/volumes"
Nov 03 20:36:23.357349 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:23.357328    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75752636-c0d6-46d6-b2d2-da4122853fac" path="/var/lib/kubelet/pods/75752636-c0d6-46d6-b2d2-da4122853fac/volumes"
Nov 03 20:36:24.047449 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:24.047411    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"]
Nov 03 20:36:24.047737 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:24.047706    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f" podUID="7aff0e54-0f19-41b1-af95-70aee78fea74" containerName="istio-proxy" containerID="cri-o://87b15fa1df7d5891154e206f57ada0197c88be773ef7beb228a0eda2465bca27" gracePeriod=30
Nov 03 20:36:27.399541 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:27.399500    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"]
Nov 03 20:36:27.404145 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:27.404114    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"]
Nov 03 20:36:27.424849 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:27.424814    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"]
Nov 03 20:36:28.731718 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:36:28.731619    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:36:29.368466 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.368439    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:36:29.396723 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.396700    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:36:29.494242 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.494182    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-envoy\") pod \"7aff0e54-0f19-41b1-af95-70aee78fea74\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") "
Nov 03 20:36:29.494426 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.494264    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-workload-socket\") pod \"7aff0e54-0f19-41b1-af95-70aee78fea74\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") "
Nov 03 20:36:29.494426 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.494287    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aff0e54-0f19-41b1-af95-70aee78fea74-kube-api-access-5qdn5\" (UniqueName: \"kubernetes.io/projected/7aff0e54-0f19-41b1-af95-70aee78fea74-kube-api-access-5qdn5\") pod \"7aff0e54-0f19-41b1-af95-70aee78fea74\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") "
Nov 03 20:36:29.494426 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.494361    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aff0e54-0f19-41b1-af95-70aee78fea74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aff0e54-0f19-41b1-af95-70aee78fea74-istiod-ca-cert\") pod \"7aff0e54-0f19-41b1-af95-70aee78fea74\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") "
Nov 03 20:36:29.494426 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.494397    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-token\" (UniqueName: \"kubernetes.io/projected/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-token\") pod \"7aff0e54-0f19-41b1-af95-70aee78fea74\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") "
Nov 03 20:36:29.494672 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.494432    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-podinfo\") pod \"7aff0e54-0f19-41b1-af95-70aee78fea74\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") "
Nov 03 20:36:29.494672 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.494464    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-workload-certs\") pod \"7aff0e54-0f19-41b1-af95-70aee78fea74\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") "
Nov 03 20:36:29.494672 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.494595    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-workload-socket" pod "7aff0e54-0f19-41b1-af95-70aee78fea74" (UID: "7aff0e54-0f19-41b1-af95-70aee78fea74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:29.494847 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.494772    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-data\") pod \"7aff0e54-0f19-41b1-af95-70aee78fea74\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") "
Nov 03 20:36:29.494847 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.494801    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aff0e54-0f19-41b1-af95-70aee78fea74-istiod-ca-cert" pod "7aff0e54-0f19-41b1-af95-70aee78fea74" (UID: "7aff0e54-0f19-41b1-af95-70aee78fea74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:36:29.494962 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.494848    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-credential-socket\") pod \"7aff0e54-0f19-41b1-af95-70aee78fea74\" (UID: \"7aff0e54-0f19-41b1-af95-70aee78fea74\") "
Nov 03 20:36:29.494962 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.494918    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-workload-certs" pod "7aff0e54-0f19-41b1-af95-70aee78fea74" (UID: "7aff0e54-0f19-41b1-af95-70aee78fea74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:29.495183 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.495127    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-data" pod "7aff0e54-0f19-41b1-af95-70aee78fea74" (UID: "7aff0e54-0f19-41b1-af95-70aee78fea74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:29.495432 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.495408    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-credential-socket" pod "7aff0e54-0f19-41b1-af95-70aee78fea74" (UID: "7aff0e54-0f19-41b1-af95-70aee78fea74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:29.496019 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.495998    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:29.496060 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.496028    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:29.496060 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.496043    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:29.496124 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.496057    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aff0e54-0f19-41b1-af95-70aee78fea74-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:29.496124 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.496072    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:29.496897 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.496873    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-token" pod "7aff0e54-0f19-41b1-af95-70aee78fea74" (UID: "7aff0e54-0f19-41b1-af95-70aee78fea74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:29.496971 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.496946    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aff0e54-0f19-41b1-af95-70aee78fea74-kube-api-access-5qdn5" pod "7aff0e54-0f19-41b1-af95-70aee78fea74" (UID: "7aff0e54-0f19-41b1-af95-70aee78fea74"). InnerVolumeSpecName "kube-api-access-5qdn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:29.497268 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.497230    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-podinfo" pod "7aff0e54-0f19-41b1-af95-70aee78fea74" (UID: "7aff0e54-0f19-41b1-af95-70aee78fea74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:36:29.497268 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.497248    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-envoy" pod "7aff0e54-0f19-41b1-af95-70aee78fea74" (UID: "7aff0e54-0f19-41b1-af95-70aee78fea74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:29.597890 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.597849    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:29.597890 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.597889    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qdn5\" (UniqueName: \"kubernetes.io/projected/7aff0e54-0f19-41b1-af95-70aee78fea74-kube-api-access-5qdn5\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:29.598130 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.597904    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:29.598130 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.597916    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aff0e54-0f19-41b1-af95-70aee78fea74-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:29.872179 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.872141    5591 generic.go:356] "Generic (PLEG): container finished" podID="7aff0e54-0f19-41b1-af95-70aee78fea74" containerID="87b15fa1df7d5891154e206f57ada0197c88be773ef7beb228a0eda2465bca27" exitCode=0
Nov 03 20:36:29.872597 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.872244    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"
Nov 03 20:36:29.872597 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.872241    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f" event={"ID":"7aff0e54-0f19-41b1-af95-70aee78fea74","Type":"ContainerDied","Data":"87b15fa1df7d5891154e206f57ada0197c88be773ef7beb228a0eda2465bca27"}
Nov 03 20:36:29.872597 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.872393    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f" event={"ID":"7aff0e54-0f19-41b1-af95-70aee78fea74","Type":"ContainerDied","Data":"e12a74a306289aea3aa9f7d44645922676a293a5a8966863925a957373ce3af2"}
Nov 03 20:36:29.872597 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.872417    5591 scope.go:122] "RemoveContainer" containerID="87b15fa1df7d5891154e206f57ada0197c88be773ef7beb228a0eda2465bca27"
Nov 03 20:36:29.885960 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.885935    5591 scope.go:122] "RemoveContainer" containerID="519e0c1283e7fc13fbf46f1e602664f635ae39749f0aae6bbb1f164e4a7f884f"
Nov 03 20:36:29.897743 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.897712    5591 scope.go:122] "RemoveContainer" containerID="87b15fa1df7d5891154e206f57ada0197c88be773ef7beb228a0eda2465bca27"
Nov 03 20:36:29.898060 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:36:29.898027    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b15fa1df7d5891154e206f57ada0197c88be773ef7beb228a0eda2465bca27\": container with ID starting with 87b15fa1df7d5891154e206f57ada0197c88be773ef7beb228a0eda2465bca27 not found: ID does not exist" containerID="87b15fa1df7d5891154e206f57ada0197c88be773ef7beb228a0eda2465bca27"
Nov 03 20:36:29.898123 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.898067    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b15fa1df7d5891154e206f57ada0197c88be773ef7beb228a0eda2465bca27"} err="failed to get container status \"87b15fa1df7d5891154e206f57ada0197c88be773ef7beb228a0eda2465bca27\": rpc error: code = NotFound desc = could not find container \"87b15fa1df7d5891154e206f57ada0197c88be773ef7beb228a0eda2465bca27\": container with ID starting with 87b15fa1df7d5891154e206f57ada0197c88be773ef7beb228a0eda2465bca27 not found: ID does not exist"
Nov 03 20:36:29.898123 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.898091    5591 scope.go:122] "RemoveContainer" containerID="519e0c1283e7fc13fbf46f1e602664f635ae39749f0aae6bbb1f164e4a7f884f"
Nov 03 20:36:29.898391 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:36:29.898371    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"519e0c1283e7fc13fbf46f1e602664f635ae39749f0aae6bbb1f164e4a7f884f\": container with ID starting with 519e0c1283e7fc13fbf46f1e602664f635ae39749f0aae6bbb1f164e4a7f884f not found: ID does not exist" containerID="519e0c1283e7fc13fbf46f1e602664f635ae39749f0aae6bbb1f164e4a7f884f"
Nov 03 20:36:29.898456 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.898407    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"519e0c1283e7fc13fbf46f1e602664f635ae39749f0aae6bbb1f164e4a7f884f"} err="failed to get container status \"519e0c1283e7fc13fbf46f1e602664f635ae39749f0aae6bbb1f164e4a7f884f\": rpc error: code = NotFound desc = could not find container \"519e0c1283e7fc13fbf46f1e602664f635ae39749f0aae6bbb1f164e4a7f884f\": container with ID starting with 519e0c1283e7fc13fbf46f1e602664f635ae39749f0aae6bbb1f164e4a7f884f not found: ID does not exist"
Nov 03 20:36:29.900442 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.900416    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"]
Nov 03 20:36:29.905866 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:29.905843    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4fb1d0-openshift-default-85c7dddd6f-56f6f"]
Nov 03 20:36:31.352405 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:31.352372    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aff0e54-0f19-41b1-af95-70aee78fea74" path="/var/lib/kubelet/pods/7aff0e54-0f19-41b1-af95-70aee78fea74/volumes"
Nov 03 20:36:31.845438 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:31.845403    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm"]
Nov 03 20:36:31.863432 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:31.863398    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc"]
Nov 03 20:36:36.858827 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:36.858781    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm" podUID="150e0fa8-4c32-418d-ae65-384d1baf47b9" containerName="payload-processing" containerID="cri-o://027a82223e775ceaf8f525069da2e28ecb42247d963bd82aa5c707eda4fe870c" gracePeriod=25
Nov 03 20:36:36.870511 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:36.870363    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc" podUID="d67ada42-276d-41e3-bb13-05a8bd3d1029" containerName="payload-pre-processing" containerID="cri-o://712946f2423f14f6ea086b6f037db4354d3a0893015cb4c1a4a87366306b8e48" gracePeriod=25
Nov 03 20:36:37.220855 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:37.220825    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc"
Nov 03 20:36:37.225847 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:37.225826    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm"
Nov 03 20:36:37.277941 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:37.277907    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d67ada42-276d-41e3-bb13-05a8bd3d1029-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67ada42-276d-41e3-bb13-05a8bd3d1029-plugins-config-volume\") pod \"d67ada42-276d-41e3-bb13-05a8bd3d1029\" (UID: \"d67ada42-276d-41e3-bb13-05a8bd3d1029\") "
Nov 03 20:36:37.278278 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:37.278257    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d67ada42-276d-41e3-bb13-05a8bd3d1029-kube-api-access-7lxbb\" (UniqueName: \"kubernetes.io/projected/d67ada42-276d-41e3-bb13-05a8bd3d1029-kube-api-access-7lxbb\") pod \"d67ada42-276d-41e3-bb13-05a8bd3d1029\" (UID: \"d67ada42-276d-41e3-bb13-05a8bd3d1029\") "
Nov 03 20:36:37.278408 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:37.278317    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67ada42-276d-41e3-bb13-05a8bd3d1029-plugins-config-volume" pod "d67ada42-276d-41e3-bb13-05a8bd3d1029" (UID: "d67ada42-276d-41e3-bb13-05a8bd3d1029"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:36:37.278408 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:37.278346    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/150e0fa8-4c32-418d-ae65-384d1baf47b9-kube-api-access-7l6zp\" (UniqueName: \"kubernetes.io/projected/150e0fa8-4c32-418d-ae65-384d1baf47b9-kube-api-access-7l6zp\") pod \"150e0fa8-4c32-418d-ae65-384d1baf47b9\" (UID: \"150e0fa8-4c32-418d-ae65-384d1baf47b9\") "
Nov 03 20:36:37.278408 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:37.278402    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/150e0fa8-4c32-418d-ae65-384d1baf47b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/150e0fa8-4c32-418d-ae65-384d1baf47b9-plugins-config-volume\") pod \"150e0fa8-4c32-418d-ae65-384d1baf47b9\" (UID: \"150e0fa8-4c32-418d-ae65-384d1baf47b9\") "
Nov 03 20:36:37.278850 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:37.278825    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/150e0fa8-4c32-418d-ae65-384d1baf47b9-plugins-config-volume" pod "150e0fa8-4c32-418d-ae65-384d1baf47b9" (UID: "150e0fa8-4c32-418d-ae65-384d1baf47b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:36:37.279288 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:37.279268    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/150e0fa8-4c32-418d-ae65-384d1baf47b9-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:37.279384 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:37.279292    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d67ada42-276d-41e3-bb13-05a8bd3d1029-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:37.281154 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:37.281124    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67ada42-276d-41e3-bb13-05a8bd3d1029-kube-api-access-7lxbb" pod "d67ada42-276d-41e3-bb13-05a8bd3d1029" (UID: "d67ada42-276d-41e3-bb13-05a8bd3d1029"). InnerVolumeSpecName "kube-api-access-7lxbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:37.281305 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:37.281281    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/150e0fa8-4c32-418d-ae65-384d1baf47b9-kube-api-access-7l6zp" pod "150e0fa8-4c32-418d-ae65-384d1baf47b9" (UID: "150e0fa8-4c32-418d-ae65-384d1baf47b9"). InnerVolumeSpecName "kube-api-access-7l6zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:37.380703 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:37.380672    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lxbb\" (UniqueName: \"kubernetes.io/projected/d67ada42-276d-41e3-bb13-05a8bd3d1029-kube-api-access-7lxbb\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:37.380703 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:37.380700    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l6zp\" (UniqueName: \"kubernetes.io/projected/150e0fa8-4c32-418d-ae65-384d1baf47b9-kube-api-access-7l6zp\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:37.457021 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:37.456903    5591 scope.go:122] "RemoveContainer" containerID="712946f2423f14f6ea086b6f037db4354d3a0893015cb4c1a4a87366306b8e48"
Nov 03 20:36:37.468840 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:37.468816    5591 scope.go:122] "RemoveContainer" containerID="027a82223e775ceaf8f525069da2e28ecb42247d963bd82aa5c707eda4fe870c"
Nov 03 20:36:37.946041 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:37.945998    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm" event={"ID":"150e0fa8-4c32-418d-ae65-384d1baf47b9","Type":"ContainerDied","Data":"027a82223e775ceaf8f525069da2e28ecb42247d963bd82aa5c707eda4fe870c"}
Nov 03 20:36:37.946513 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:37.946084    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm" event={"ID":"150e0fa8-4c32-418d-ae65-384d1baf47b9","Type":"ContainerDied","Data":"3ba13573d7804eec68435d85e9a32248329a9ca6b6ddc6ff687d8fcd5f323f80"}
Nov 03 20:36:37.946513 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:37.946103    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc" event={"ID":"d67ada42-276d-41e3-bb13-05a8bd3d1029","Type":"ContainerDied","Data":"712946f2423f14f6ea086b6f037db4354d3a0893015cb4c1a4a87366306b8e48"}
Nov 03 20:36:37.946513 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:37.946121    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc" event={"ID":"d67ada42-276d-41e3-bb13-05a8bd3d1029","Type":"ContainerDied","Data":"b74131792ba217a0bfe82d5e349d53055f2941a09e1ef5979822479ef07b3b85"}
Nov 03 20:36:37.946513 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:37.946142    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm"
Nov 03 20:36:37.946513 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:37.946142    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc"
Nov 03 20:36:38.000264 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:37.999862    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc"]
Nov 03 20:36:38.003998 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:38.003973    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-81a5ae-84ff5c8d59-gtfbc"]
Nov 03 20:36:38.014505 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:38.014476    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm"]
Nov 03 20:36:38.019136 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:38.019111    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-81a5ae-9c66fb5b-nqjxm"]
Nov 03 20:36:39.352090 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:39.352046    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="150e0fa8-4c32-418d-ae65-384d1baf47b9" path="/var/lib/kubelet/pods/150e0fa8-4c32-418d-ae65-384d1baf47b9/volumes"
Nov 03 20:36:39.352771 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:39.352748    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d67ada42-276d-41e3-bb13-05a8bd3d1029" path="/var/lib/kubelet/pods/d67ada42-276d-41e3-bb13-05a8bd3d1029/volumes"
Nov 03 20:36:39.715276 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:39.714339    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6"]
Nov 03 20:36:39.715276 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:39.715072    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6" podUID="9a4ae1ca-060a-43e4-b877-02a2c3565899" containerName="maas-api" containerID="cri-o://1359ce024772bfee9df5dc69fcf03c0a8aeaba076480724d339a894e23a8a5cf" gracePeriod=30
Nov 03 20:36:40.170574 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:40.169180    5591 generic.go:356] "Generic (PLEG): container finished" podID="9a4ae1ca-060a-43e4-b877-02a2c3565899" containerID="1359ce024772bfee9df5dc69fcf03c0a8aeaba076480724d339a894e23a8a5cf" exitCode=0
Nov 03 20:36:40.170574 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:40.170134    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6" event={"ID":"9a4ae1ca-060a-43e4-b877-02a2c3565899","Type":"ContainerDied","Data":"1359ce024772bfee9df5dc69fcf03c0a8aeaba076480724d339a894e23a8a5cf"}
Nov 03 20:36:40.224191 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:40.224156    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6"
Nov 03 20:36:40.289451 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:40.289420    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a4ae1ca-060a-43e4-b877-02a2c3565899-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4ae1ca-060a-43e4-b877-02a2c3565899-openshift-service-ca\") pod \"9a4ae1ca-060a-43e4-b877-02a2c3565899\" (UID: \"9a4ae1ca-060a-43e4-b877-02a2c3565899\") "
Nov 03 20:36:40.289665 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:40.289498    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a4ae1ca-060a-43e4-b877-02a2c3565899-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4ae1ca-060a-43e4-b877-02a2c3565899-metrics-certs\") pod \"9a4ae1ca-060a-43e4-b877-02a2c3565899\" (UID: \"9a4ae1ca-060a-43e4-b877-02a2c3565899\") "
Nov 03 20:36:40.289665 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:40.289625    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a4ae1ca-060a-43e4-b877-02a2c3565899-kube-api-access-zvj8c\" (UniqueName: \"kubernetes.io/projected/9a4ae1ca-060a-43e4-b877-02a2c3565899-kube-api-access-zvj8c\") pod \"9a4ae1ca-060a-43e4-b877-02a2c3565899\" (UID: \"9a4ae1ca-060a-43e4-b877-02a2c3565899\") "
Nov 03 20:36:40.289795 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:40.289778    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a4ae1ca-060a-43e4-b877-02a2c3565899-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a4ae1ca-060a-43e4-b877-02a2c3565899-maas-api-tls\") pod \"9a4ae1ca-060a-43e4-b877-02a2c3565899\" (UID: \"9a4ae1ca-060a-43e4-b877-02a2c3565899\") "
Nov 03 20:36:40.289954 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:40.289913    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a4ae1ca-060a-43e4-b877-02a2c3565899-openshift-service-ca" pod "9a4ae1ca-060a-43e4-b877-02a2c3565899" (UID: "9a4ae1ca-060a-43e4-b877-02a2c3565899"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:36:40.290736 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:40.290705    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4ae1ca-060a-43e4-b877-02a2c3565899-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:40.294171 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:40.292655    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a4ae1ca-060a-43e4-b877-02a2c3565899-metrics-certs" pod "9a4ae1ca-060a-43e4-b877-02a2c3565899" (UID: "9a4ae1ca-060a-43e4-b877-02a2c3565899"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:40.294367 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:40.294191    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a4ae1ca-060a-43e4-b877-02a2c3565899-kube-api-access-zvj8c" pod "9a4ae1ca-060a-43e4-b877-02a2c3565899" (UID: "9a4ae1ca-060a-43e4-b877-02a2c3565899"). InnerVolumeSpecName "kube-api-access-zvj8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:40.297733 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:40.297705    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a4ae1ca-060a-43e4-b877-02a2c3565899-maas-api-tls" pod "9a4ae1ca-060a-43e4-b877-02a2c3565899" (UID: "9a4ae1ca-060a-43e4-b877-02a2c3565899"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:40.392224 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:40.392165    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4ae1ca-060a-43e4-b877-02a2c3565899-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:40.392224 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:40.392199    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvj8c\" (UniqueName: \"kubernetes.io/projected/9a4ae1ca-060a-43e4-b877-02a2c3565899-kube-api-access-zvj8c\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:40.392735 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:40.392235    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a4ae1ca-060a-43e4-b877-02a2c3565899-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:41.185279 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:41.185244    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6" event={"ID":"9a4ae1ca-060a-43e4-b877-02a2c3565899","Type":"ContainerDied","Data":"569ea883e14c0aeb2a2d0c55216068a476d83ed03fe521eea2202ee826d30860"}
Nov 03 20:36:41.185279 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:41.185264    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6"
Nov 03 20:36:41.185523 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:41.185305    5591 scope.go:122] "RemoveContainer" containerID="1359ce024772bfee9df5dc69fcf03c0a8aeaba076480724d339a894e23a8a5cf"
Nov 03 20:36:41.213915 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:41.213886    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6"]
Nov 03 20:36:41.219040 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:41.219017    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-81a5ae-67f5694457-hwhn6"]
Nov 03 20:36:41.354643 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:41.354612    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a4ae1ca-060a-43e4-b877-02a2c3565899" path="/var/lib/kubelet/pods/9a4ae1ca-060a-43e4-b877-02a2c3565899/volumes"
Nov 03 20:36:42.445229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.440692    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" containerID="cri-o://87813c228798aea130e8d2e12cac5b59a3533e7b8d7a8453209989e8d6ac0d70" gracePeriod=45
Nov 03 20:36:42.445229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.440879    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main" containerID="cri-o://920db3ea2196aacbe6905c33c7084feccf0c5a5d2e1db64577815bcf2c60e1ab" gracePeriod=45
Nov 03 20:36:42.460795 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.460741    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46" podUID="ea3c9bb0-fa05-4815-8412-6db003469cd0" containerName="main" containerID="cri-o://d67af5494e52ed57f05c266adfb74510944c6479d0ee6ccd5adffd1ff21a4513" gracePeriod=45
Nov 03 20:36:42.500228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.500182    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": dial tcp 10.134.0.69:8000: connect: connection refused"
Nov 03 20:36:42.701011 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.700907    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46" podUID="ea3c9bb0-fa05-4815-8412-6db003469cd0" containerName="main" probeResult="failure" output="Get \"https://10.134.1.8:8000/ready\": dial tcp 10.134.1.8:8000: connect: connection refused"
Nov 03 20:36:42.803101 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.803075    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 20:36:42.807906 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.807872    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 20:36:42.852685 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.852652    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-model-cache\") pod \"2c1b25f8-ea79-44e9-991f-1067e71cf960\" (UID: \"2c1b25f8-ea79-44e9-991f-1067e71cf960\") "
Nov 03 20:36:42.852981 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.852954    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-tmp-dir\") pod \"2c1b25f8-ea79-44e9-991f-1067e71cf960\" (UID: \"2c1b25f8-ea79-44e9-991f-1067e71cf960\") "
Nov 03 20:36:42.853107 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.852989    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-dshm\") pod \"2c1b25f8-ea79-44e9-991f-1067e71cf960\" (UID: \"2c1b25f8-ea79-44e9-991f-1067e71cf960\") "
Nov 03 20:36:42.853107 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.853025    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-tls-certs\" (UniqueName: \"kubernetes.io/secret/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-tls-certs\") pod \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\" (UID: \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\") "
Nov 03 20:36:42.853107 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.853062    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-tmp-dir\") pod \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\" (UID: \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\") "
Nov 03 20:36:42.853107 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.853078    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c1b25f8-ea79-44e9-991f-1067e71cf960-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c1b25f8-ea79-44e9-991f-1067e71cf960-tls-certs\") pod \"2c1b25f8-ea79-44e9-991f-1067e71cf960\" (UID: \"2c1b25f8-ea79-44e9-991f-1067e71cf960\") "
Nov 03 20:36:42.853107 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.853104    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-home\" (UniqueName: \"kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-home\") pod \"2c1b25f8-ea79-44e9-991f-1067e71cf960\" (UID: \"2c1b25f8-ea79-44e9-991f-1067e71cf960\") "
Nov 03 20:36:42.853432 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.853131    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-kube-api-access-cfpht\" (UniqueName: \"kubernetes.io/projected/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-kube-api-access-cfpht\") pod \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\" (UID: \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\") "
Nov 03 20:36:42.853432 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.853166    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c1b25f8-ea79-44e9-991f-1067e71cf960-kube-api-access-mhlkk\" (UniqueName: \"kubernetes.io/projected/2c1b25f8-ea79-44e9-991f-1067e71cf960-kube-api-access-mhlkk\") pod \"2c1b25f8-ea79-44e9-991f-1067e71cf960\" (UID: \"2c1b25f8-ea79-44e9-991f-1067e71cf960\") "
Nov 03 20:36:42.853432 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.853250    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-home\" (UniqueName: \"kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-home\") pod \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\" (UID: \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\") "
Nov 03 20:36:42.853432 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.853326    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-dshm\") pod \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\" (UID: \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\") "
Nov 03 20:36:42.853432 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.853367    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-model-cache\") pod \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\" (UID: \"dc613fc8-12f6-4cfb-9ab1-3a112eaced51\") "
Nov 03 20:36:42.854176 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.854127    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-model-cache" pod "dc613fc8-12f6-4cfb-9ab1-3a112eaced51" (UID: "dc613fc8-12f6-4cfb-9ab1-3a112eaced51"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:42.854176 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.854134    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"
Nov 03 20:36:42.854639 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.854395    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-home" pod "dc613fc8-12f6-4cfb-9ab1-3a112eaced51" (UID: "dc613fc8-12f6-4cfb-9ab1-3a112eaced51"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:42.854639 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.854499    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-model-cache" pod "2c1b25f8-ea79-44e9-991f-1067e71cf960" (UID: "2c1b25f8-ea79-44e9-991f-1067e71cf960"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:42.854763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.854727    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-home" pod "2c1b25f8-ea79-44e9-991f-1067e71cf960" (UID: "2c1b25f8-ea79-44e9-991f-1067e71cf960"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:42.856991 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.855604    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-tmp-dir" pod "dc613fc8-12f6-4cfb-9ab1-3a112eaced51" (UID: "dc613fc8-12f6-4cfb-9ab1-3a112eaced51"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:42.856991 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.855868    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-tmp-dir" pod "2c1b25f8-ea79-44e9-991f-1067e71cf960" (UID: "2c1b25f8-ea79-44e9-991f-1067e71cf960"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:42.860394 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.860350    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c1b25f8-ea79-44e9-991f-1067e71cf960-kube-api-access-mhlkk" pod "2c1b25f8-ea79-44e9-991f-1067e71cf960" (UID: "2c1b25f8-ea79-44e9-991f-1067e71cf960"). InnerVolumeSpecName "kube-api-access-mhlkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:42.860985 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.860949    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-kube-api-access-cfpht" pod "dc613fc8-12f6-4cfb-9ab1-3a112eaced51" (UID: "dc613fc8-12f6-4cfb-9ab1-3a112eaced51"). InnerVolumeSpecName "kube-api-access-cfpht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:42.861720 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.861675    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-dshm" pod "2c1b25f8-ea79-44e9-991f-1067e71cf960" (UID: "2c1b25f8-ea79-44e9-991f-1067e71cf960"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:42.863857 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.863830    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-dshm" pod "dc613fc8-12f6-4cfb-9ab1-3a112eaced51" (UID: "dc613fc8-12f6-4cfb-9ab1-3a112eaced51"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:42.865478 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.865420    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-tls-certs" pod "dc613fc8-12f6-4cfb-9ab1-3a112eaced51" (UID: "dc613fc8-12f6-4cfb-9ab1-3a112eaced51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:42.874590 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.874533    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c1b25f8-ea79-44e9-991f-1067e71cf960-tls-certs" pod "2c1b25f8-ea79-44e9-991f-1067e71cf960" (UID: "2c1b25f8-ea79-44e9-991f-1067e71cf960"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:42.955113 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.955008    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-home\" (UniqueName: \"kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-home\") pod \"ea3c9bb0-fa05-4815-8412-6db003469cd0\" (UID: \"ea3c9bb0-fa05-4815-8412-6db003469cd0\") "
Nov 03 20:36:42.955318 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.955151    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea3c9bb0-fa05-4815-8412-6db003469cd0-tls-certs\" (UniqueName: \"kubernetes.io/secret/ea3c9bb0-fa05-4815-8412-6db003469cd0-tls-certs\") pod \"ea3c9bb0-fa05-4815-8412-6db003469cd0\" (UID: \"ea3c9bb0-fa05-4815-8412-6db003469cd0\") "
Nov 03 20:36:42.955318 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.955188    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-model-cache\") pod \"ea3c9bb0-fa05-4815-8412-6db003469cd0\" (UID: \"ea3c9bb0-fa05-4815-8412-6db003469cd0\") "
Nov 03 20:36:42.955318 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.955303    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea3c9bb0-fa05-4815-8412-6db003469cd0-kube-api-access-m4bgt\" (UniqueName: \"kubernetes.io/projected/ea3c9bb0-fa05-4815-8412-6db003469cd0-kube-api-access-m4bgt\") pod \"ea3c9bb0-fa05-4815-8412-6db003469cd0\" (UID: \"ea3c9bb0-fa05-4815-8412-6db003469cd0\") "
Nov 03 20:36:42.955490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.955335    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-tmp-dir\") pod \"ea3c9bb0-fa05-4815-8412-6db003469cd0\" (UID: \"ea3c9bb0-fa05-4815-8412-6db003469cd0\") "
Nov 03 20:36:42.955490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.955374    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-home" pod "ea3c9bb0-fa05-4815-8412-6db003469cd0" (UID: "ea3c9bb0-fa05-4815-8412-6db003469cd0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:42.955490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.955468    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-dshm\") pod \"ea3c9bb0-fa05-4815-8412-6db003469cd0\" (UID: \"ea3c9bb0-fa05-4815-8412-6db003469cd0\") "
Nov 03 20:36:42.955641 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.955598    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-model-cache" pod "ea3c9bb0-fa05-4815-8412-6db003469cd0" (UID: "ea3c9bb0-fa05-4815-8412-6db003469cd0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:42.956401 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.955841    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-tmp-dir" pod "ea3c9bb0-fa05-4815-8412-6db003469cd0" (UID: "ea3c9bb0-fa05-4815-8412-6db003469cd0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:42.956401 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.956159    5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-tmp-dir\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:42.956401 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.956182    5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c1b25f8-ea79-44e9-991f-1067e71cf960-tls-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:42.956401 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.956196    5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-home\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:42.956401 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.956230    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfpht\" (UniqueName: \"kubernetes.io/projected/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-kube-api-access-cfpht\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:42.956401 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.956246    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhlkk\" (UniqueName: \"kubernetes.io/projected/2c1b25f8-ea79-44e9-991f-1067e71cf960-kube-api-access-mhlkk\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:42.956401 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.956260    5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-model-cache\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:42.956401 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.956274    5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-home\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:42.956401 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.956288    5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-tmp-dir\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:42.956401 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.956300    5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-dshm\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:42.956401 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.956312    5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-model-cache\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:42.956401 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.956327    5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-model-cache\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:42.956401 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.956340    5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-home\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:42.956401 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.956352    5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-tmp-dir\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:42.956401 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.956364    5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c1b25f8-ea79-44e9-991f-1067e71cf960-dshm\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:42.956401 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.956376    5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc613fc8-12f6-4cfb-9ab1-3a112eaced51-tls-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:42.958313 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.958285    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea3c9bb0-fa05-4815-8412-6db003469cd0-tls-certs" pod "ea3c9bb0-fa05-4815-8412-6db003469cd0" (UID: "ea3c9bb0-fa05-4815-8412-6db003469cd0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:36:42.958499 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.958481    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3c9bb0-fa05-4815-8412-6db003469cd0-kube-api-access-m4bgt" pod "ea3c9bb0-fa05-4815-8412-6db003469cd0" (UID: "ea3c9bb0-fa05-4815-8412-6db003469cd0"). InnerVolumeSpecName "kube-api-access-m4bgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:36:42.958573 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:42.958494    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-dshm" pod "ea3c9bb0-fa05-4815-8412-6db003469cd0" (UID: "ea3c9bb0-fa05-4815-8412-6db003469cd0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:36:43.073230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.067640    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4bgt\" (UniqueName: \"kubernetes.io/projected/ea3c9bb0-fa05-4815-8412-6db003469cd0-kube-api-access-m4bgt\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:43.073230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.067685    5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea3c9bb0-fa05-4815-8412-6db003469cd0-dshm\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:43.073230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.067700    5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea3c9bb0-fa05-4815-8412-6db003469cd0-tls-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:36:43.227277 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.227150    5591 generic.go:356] "Generic (PLEG): container finished" podID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerID="87813c228798aea130e8d2e12cac5b59a3533e7b8d7a8453209989e8d6ac0d70" exitCode=0
Nov 03 20:36:43.227277 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.227272    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" event={"ID":"dc613fc8-12f6-4cfb-9ab1-3a112eaced51","Type":"ContainerDied","Data":"87813c228798aea130e8d2e12cac5b59a3533e7b8d7a8453209989e8d6ac0d70"}
Nov 03 20:36:43.227477 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.227302    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w" event={"ID":"dc613fc8-12f6-4cfb-9ab1-3a112eaced51","Type":"ContainerDied","Data":"1efb3380a78c9bdbbecce00892c93e7fddbbe490132d6aa03c2ec1fd3969c742"}
Nov 03 20:36:43.227477 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.227320    5591 scope.go:122] "RemoveContainer" containerID="87813c228798aea130e8d2e12cac5b59a3533e7b8d7a8453209989e8d6ac0d70"
Nov 03 20:36:43.227477 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.227420    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"
Nov 03 20:36:43.230733 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.230688    5591 generic.go:356] "Generic (PLEG): container finished" podID="ea3c9bb0-fa05-4815-8412-6db003469cd0" containerID="d67af5494e52ed57f05c266adfb74510944c6479d0ee6ccd5adffd1ff21a4513" exitCode=0
Nov 03 20:36:43.230884 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.230742    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"
Nov 03 20:36:43.230884 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.230747    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46" event={"ID":"ea3c9bb0-fa05-4815-8412-6db003469cd0","Type":"ContainerDied","Data":"d67af5494e52ed57f05c266adfb74510944c6479d0ee6ccd5adffd1ff21a4513"}
Nov 03 20:36:43.230884 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.230800    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46" event={"ID":"ea3c9bb0-fa05-4815-8412-6db003469cd0","Type":"ContainerDied","Data":"1ed12399ca1568995adebd2f8a5c29efa4f2e627d7819d02d2ce0cc1d820f684"}
Nov 03 20:36:43.233867 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.233836    5591 generic.go:356] "Generic (PLEG): container finished" podID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerID="920db3ea2196aacbe6905c33c7084feccf0c5a5d2e1db64577815bcf2c60e1ab" exitCode=0
Nov 03 20:36:43.233971 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.233954    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" event={"ID":"2c1b25f8-ea79-44e9-991f-1067e71cf960","Type":"ContainerDied","Data":"920db3ea2196aacbe6905c33c7084feccf0c5a5d2e1db64577815bcf2c60e1ab"}
Nov 03 20:36:43.234014 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.233984    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x" event={"ID":"2c1b25f8-ea79-44e9-991f-1067e71cf960","Type":"ContainerDied","Data":"f635f6627a4271c66ca3a103d7e5622ca157d54bb44ce0239265f9a75bca3c99"}
Nov 03 20:36:43.234058 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.234045    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"
Nov 03 20:36:43.244432 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.244409    5591 scope.go:122] "RemoveContainer" containerID="87813c228798aea130e8d2e12cac5b59a3533e7b8d7a8453209989e8d6ac0d70"
Nov 03 20:36:43.244921 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:36:43.244899    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87813c228798aea130e8d2e12cac5b59a3533e7b8d7a8453209989e8d6ac0d70\": container with ID starting with 87813c228798aea130e8d2e12cac5b59a3533e7b8d7a8453209989e8d6ac0d70 not found: ID does not exist" containerID="87813c228798aea130e8d2e12cac5b59a3533e7b8d7a8453209989e8d6ac0d70"
Nov 03 20:36:43.245016 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.244933    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87813c228798aea130e8d2e12cac5b59a3533e7b8d7a8453209989e8d6ac0d70"} err="failed to get container status \"87813c228798aea130e8d2e12cac5b59a3533e7b8d7a8453209989e8d6ac0d70\": rpc error: code = NotFound desc = could not find container \"87813c228798aea130e8d2e12cac5b59a3533e7b8d7a8453209989e8d6ac0d70\": container with ID starting with 87813c228798aea130e8d2e12cac5b59a3533e7b8d7a8453209989e8d6ac0d70 not found: ID does not exist"
Nov 03 20:36:43.245016 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.244955    5591 scope.go:122] "RemoveContainer" containerID="d67af5494e52ed57f05c266adfb74510944c6479d0ee6ccd5adffd1ff21a4513"
Nov 03 20:36:43.256967 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.256944    5591 scope.go:122] "RemoveContainer" containerID="d67af5494e52ed57f05c266adfb74510944c6479d0ee6ccd5adffd1ff21a4513"
Nov 03 20:36:43.257296 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:36:43.257275    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d67af5494e52ed57f05c266adfb74510944c6479d0ee6ccd5adffd1ff21a4513\": container with ID starting with d67af5494e52ed57f05c266adfb74510944c6479d0ee6ccd5adffd1ff21a4513 not found: ID does not exist" containerID="d67af5494e52ed57f05c266adfb74510944c6479d0ee6ccd5adffd1ff21a4513"
Nov 03 20:36:43.257350 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.257304    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67af5494e52ed57f05c266adfb74510944c6479d0ee6ccd5adffd1ff21a4513"} err="failed to get container status \"d67af5494e52ed57f05c266adfb74510944c6479d0ee6ccd5adffd1ff21a4513\": rpc error: code = NotFound desc = could not find container \"d67af5494e52ed57f05c266adfb74510944c6479d0ee6ccd5adffd1ff21a4513\": container with ID starting with d67af5494e52ed57f05c266adfb74510944c6479d0ee6ccd5adffd1ff21a4513 not found: ID does not exist"
Nov 03 20:36:43.257350 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.257323    5591 scope.go:122] "RemoveContainer" containerID="920db3ea2196aacbe6905c33c7084feccf0c5a5d2e1db64577815bcf2c60e1ab"
Nov 03 20:36:43.257650 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.257631    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"]
Nov 03 20:36:43.263925 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.263902    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/distinct-w4-3fbb1a-kserve-69d65db78b-thc7w"]
Nov 03 20:36:43.268706 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.268685    5591 scope.go:122] "RemoveContainer" containerID="920db3ea2196aacbe6905c33c7084feccf0c5a5d2e1db64577815bcf2c60e1ab"
Nov 03 20:36:43.269060 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:36:43.269040    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"920db3ea2196aacbe6905c33c7084feccf0c5a5d2e1db64577815bcf2c60e1ab\": container with ID starting with 920db3ea2196aacbe6905c33c7084feccf0c5a5d2e1db64577815bcf2c60e1ab not found: ID does not exist" containerID="920db3ea2196aacbe6905c33c7084feccf0c5a5d2e1db64577815bcf2c60e1ab"
Nov 03 20:36:43.269143 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.269073    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"920db3ea2196aacbe6905c33c7084feccf0c5a5d2e1db64577815bcf2c60e1ab"} err="failed to get container status \"920db3ea2196aacbe6905c33c7084feccf0c5a5d2e1db64577815bcf2c60e1ab\": rpc error: code = NotFound desc = could not find container \"920db3ea2196aacbe6905c33c7084feccf0c5a5d2e1db64577815bcf2c60e1ab\": container with ID starting with 920db3ea2196aacbe6905c33c7084feccf0c5a5d2e1db64577815bcf2c60e1ab not found: ID does not exist"
Nov 03 20:36:43.274821 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.274793    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"]
Nov 03 20:36:43.280967 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.280939    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/simulator-w4-3fbb1a-kserve-7868585bbb-5rp46"]
Nov 03 20:36:43.291475 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.291223    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"]
Nov 03 20:36:43.295561 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.295538    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3fbb1a/distinct-2-w4-3fbb1a-kserve-7678677cd9-8tr6x"]
Nov 03 20:36:43.352243 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.352192    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" path="/var/lib/kubelet/pods/2c1b25f8-ea79-44e9-991f-1067e71cf960/volumes"
Nov 03 20:36:43.352888 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.352871    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" path="/var/lib/kubelet/pods/dc613fc8-12f6-4cfb-9ab1-3a112eaced51/volumes"
Nov 03 20:36:43.353327 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:43.353312    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea3c9bb0-fa05-4815-8412-6db003469cd0" path="/var/lib/kubelet/pods/ea3c9bb0-fa05-4815-8412-6db003469cd0/volumes"
Nov 03 20:36:52.635419 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:52.635320    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"]
Nov 03 20:36:58.732166 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:36:58.732124    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:36:59.792929 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:59.792888    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm"]
Nov 03 20:36:59.808256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:36:59.808223    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb"]
Nov 03 20:37:04.802277 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:04.795547    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm" podUID="a74714b9-b503-4817-b5d2-9ab74cd798a7" containerName="payload-processing" containerID="cri-o://338c0c788bc3d388964e30c0c15dd488dddeadb72df4d0b439f20aefe9f111f4" gracePeriod=25
Nov 03 20:37:04.832284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:04.828815    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb" podUID="ed913799-a594-487c-92a1-f55b50cac03c" containerName="payload-pre-processing" containerID="cri-o://ee7bc8ed15cc181aa29bef086cc5ae20e13c64754b64b2c6da1dc86594f63733" gracePeriod=25
Nov 03 20:37:05.273680 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.265403    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm"
Nov 03 20:37:05.285290 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.285264    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb"
Nov 03 20:37:05.327918 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.327880    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a74714b9-b503-4817-b5d2-9ab74cd798a7-kube-api-access-lfhdg\" (UniqueName: \"kubernetes.io/projected/a74714b9-b503-4817-b5d2-9ab74cd798a7-kube-api-access-lfhdg\") pod \"a74714b9-b503-4817-b5d2-9ab74cd798a7\" (UID: \"a74714b9-b503-4817-b5d2-9ab74cd798a7\") "
Nov 03 20:37:05.328128 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.328014    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed913799-a594-487c-92a1-f55b50cac03c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed913799-a594-487c-92a1-f55b50cac03c-plugins-config-volume\") pod \"ed913799-a594-487c-92a1-f55b50cac03c\" (UID: \"ed913799-a594-487c-92a1-f55b50cac03c\") "
Nov 03 20:37:05.328128 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.328067    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed913799-a594-487c-92a1-f55b50cac03c-kube-api-access-sp26f\" (UniqueName: \"kubernetes.io/projected/ed913799-a594-487c-92a1-f55b50cac03c-kube-api-access-sp26f\") pod \"ed913799-a594-487c-92a1-f55b50cac03c\" (UID: \"ed913799-a594-487c-92a1-f55b50cac03c\") "
Nov 03 20:37:05.328306 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.328146    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a74714b9-b503-4817-b5d2-9ab74cd798a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a74714b9-b503-4817-b5d2-9ab74cd798a7-plugins-config-volume\") pod \"a74714b9-b503-4817-b5d2-9ab74cd798a7\" (UID: \"a74714b9-b503-4817-b5d2-9ab74cd798a7\") "
Nov 03 20:37:05.328660 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.328515    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed913799-a594-487c-92a1-f55b50cac03c-plugins-config-volume" pod "ed913799-a594-487c-92a1-f55b50cac03c" (UID: "ed913799-a594-487c-92a1-f55b50cac03c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:05.329004 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.328965    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74714b9-b503-4817-b5d2-9ab74cd798a7-plugins-config-volume" pod "a74714b9-b503-4817-b5d2-9ab74cd798a7" (UID: "a74714b9-b503-4817-b5d2-9ab74cd798a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:05.331484 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.331453    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed913799-a594-487c-92a1-f55b50cac03c-kube-api-access-sp26f" pod "ed913799-a594-487c-92a1-f55b50cac03c" (UID: "ed913799-a594-487c-92a1-f55b50cac03c"). InnerVolumeSpecName "kube-api-access-sp26f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:05.336606 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.336578    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed913799-a594-487c-92a1-f55b50cac03c-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:05.336722 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.336611    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp26f\" (UniqueName: \"kubernetes.io/projected/ed913799-a594-487c-92a1-f55b50cac03c-kube-api-access-sp26f\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:05.336722 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.336663    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a74714b9-b503-4817-b5d2-9ab74cd798a7-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:05.342156 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.342127    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74714b9-b503-4817-b5d2-9ab74cd798a7-kube-api-access-lfhdg" pod "a74714b9-b503-4817-b5d2-9ab74cd798a7" (UID: "a74714b9-b503-4817-b5d2-9ab74cd798a7"). InnerVolumeSpecName "kube-api-access-lfhdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:05.416634 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.416492    5591 generic.go:356] "Generic (PLEG): container finished" podID="ed913799-a594-487c-92a1-f55b50cac03c" containerID="ee7bc8ed15cc181aa29bef086cc5ae20e13c64754b64b2c6da1dc86594f63733" exitCode=0
Nov 03 20:37:05.416865 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.416768    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb"
Nov 03 20:37:05.417506 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.417481    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb" event={"ID":"ed913799-a594-487c-92a1-f55b50cac03c","Type":"ContainerDied","Data":"ee7bc8ed15cc181aa29bef086cc5ae20e13c64754b64b2c6da1dc86594f63733"}
Nov 03 20:37:05.417642 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.417534    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb" event={"ID":"ed913799-a594-487c-92a1-f55b50cac03c","Type":"ContainerDied","Data":"7bffa589610413c4f2085f48c24761ce7fa956abe09d0925ad424b199ced7cca"}
Nov 03 20:37:05.417642 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.417563    5591 scope.go:122] "RemoveContainer" containerID="ee7bc8ed15cc181aa29bef086cc5ae20e13c64754b64b2c6da1dc86594f63733"
Nov 03 20:37:05.419442 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.419416    5591 generic.go:356] "Generic (PLEG): container finished" podID="a74714b9-b503-4817-b5d2-9ab74cd798a7" containerID="338c0c788bc3d388964e30c0c15dd488dddeadb72df4d0b439f20aefe9f111f4" exitCode=0
Nov 03 20:37:05.419604 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.419483    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm"
Nov 03 20:37:05.419688 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.419660    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm" event={"ID":"a74714b9-b503-4817-b5d2-9ab74cd798a7","Type":"ContainerDied","Data":"338c0c788bc3d388964e30c0c15dd488dddeadb72df4d0b439f20aefe9f111f4"}
Nov 03 20:37:05.419762 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.419731    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm" event={"ID":"a74714b9-b503-4817-b5d2-9ab74cd798a7","Type":"ContainerDied","Data":"c5d45575c48ef1cbb626ba17f44ba0fff510d25d9cea039f3061421aad205d7d"}
Nov 03 20:37:05.430046 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.430027    5591 scope.go:122] "RemoveContainer" containerID="ee7bc8ed15cc181aa29bef086cc5ae20e13c64754b64b2c6da1dc86594f63733"
Nov 03 20:37:05.430407 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:37:05.430385    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee7bc8ed15cc181aa29bef086cc5ae20e13c64754b64b2c6da1dc86594f63733\": container with ID starting with ee7bc8ed15cc181aa29bef086cc5ae20e13c64754b64b2c6da1dc86594f63733 not found: ID does not exist" containerID="ee7bc8ed15cc181aa29bef086cc5ae20e13c64754b64b2c6da1dc86594f63733"
Nov 03 20:37:05.430514 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.430415    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee7bc8ed15cc181aa29bef086cc5ae20e13c64754b64b2c6da1dc86594f63733"} err="failed to get container status \"ee7bc8ed15cc181aa29bef086cc5ae20e13c64754b64b2c6da1dc86594f63733\": rpc error: code = NotFound desc = could not find container \"ee7bc8ed15cc181aa29bef086cc5ae20e13c64754b64b2c6da1dc86594f63733\": container with ID starting with ee7bc8ed15cc181aa29bef086cc5ae20e13c64754b64b2c6da1dc86594f63733 not found: ID does not exist"
Nov 03 20:37:05.430514 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.430429    5591 scope.go:122] "RemoveContainer" containerID="338c0c788bc3d388964e30c0c15dd488dddeadb72df4d0b439f20aefe9f111f4"
Nov 03 20:37:05.438154 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.438127    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfhdg\" (UniqueName: \"kubernetes.io/projected/a74714b9-b503-4817-b5d2-9ab74cd798a7-kube-api-access-lfhdg\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:05.439275 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.439252    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb"]
Nov 03 20:37:05.441061 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.441038    5591 scope.go:122] "RemoveContainer" containerID="338c0c788bc3d388964e30c0c15dd488dddeadb72df4d0b439f20aefe9f111f4"
Nov 03 20:37:05.441599 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:37:05.441569    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"338c0c788bc3d388964e30c0c15dd488dddeadb72df4d0b439f20aefe9f111f4\": container with ID starting with 338c0c788bc3d388964e30c0c15dd488dddeadb72df4d0b439f20aefe9f111f4 not found: ID does not exist" containerID="338c0c788bc3d388964e30c0c15dd488dddeadb72df4d0b439f20aefe9f111f4"
Nov 03 20:37:05.441710 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.441598    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"338c0c788bc3d388964e30c0c15dd488dddeadb72df4d0b439f20aefe9f111f4"} err="failed to get container status \"338c0c788bc3d388964e30c0c15dd488dddeadb72df4d0b439f20aefe9f111f4\": rpc error: code = NotFound desc = could not find container \"338c0c788bc3d388964e30c0c15dd488dddeadb72df4d0b439f20aefe9f111f4\": container with ID starting with 338c0c788bc3d388964e30c0c15dd488dddeadb72df4d0b439f20aefe9f111f4 not found: ID does not exist"
Nov 03 20:37:05.445509 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.445486    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-d3d0df-6ff8d4b9ff-dw4kb"]
Nov 03 20:37:05.459693 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.459671    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm"]
Nov 03 20:37:05.467093 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:05.467069    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d3d0df-c85975868-kxxcm"]
Nov 03 20:37:06.844407 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:06.836396    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"]
Nov 03 20:37:06.844407 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:06.836630    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm" podUID="e31559fd-2ec2-48cd-8f00-8167a567097b" containerName="istio-proxy" containerID="cri-o://0d4e4f3915c407f348d4baf6f79380faa66d002ca74f21e14d930c0248bd2501" gracePeriod=30
Nov 03 20:37:07.353536 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:07.353506    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a74714b9-b503-4817-b5d2-9ab74cd798a7" path="/var/lib/kubelet/pods/a74714b9-b503-4817-b5d2-9ab74cd798a7/volumes"
Nov 03 20:37:07.354089 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:07.354069    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed913799-a594-487c-92a1-f55b50cac03c" path="/var/lib/kubelet/pods/ed913799-a594-487c-92a1-f55b50cac03c/volumes"
Nov 03 20:37:07.659546 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:07.659444    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main" containerID="cri-o://d03a7ea2a9efea2faa83bd80d81895629158b902359440342412831d2a55aaaf" gracePeriod=45
Nov 03 20:37:07.958350 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:07.958323    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 20:37:08.011199 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:08.011164    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-home\" (UniqueName: \"kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-home\") pod \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\" (UID: \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\") "
Nov 03 20:37:08.011405 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:08.011294    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-tmp-dir\") pod \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\" (UID: \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\") "
Nov 03 20:37:08.011485 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:08.011401    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-tls-certs\" (UniqueName: \"kubernetes.io/secret/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-tls-certs\") pod \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\" (UID: \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\") "
Nov 03 20:37:08.011485 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:08.011446    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-home" pod "7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" (UID: "7173cfa7-fb5e-40f3-9a2c-0f60ebef0862"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:08.011485 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:08.011465    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-kube-api-access-7c4cp\" (UniqueName: \"kubernetes.io/projected/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-kube-api-access-7c4cp\") pod \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\" (UID: \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\") "
Nov 03 20:37:08.011682 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:08.011505    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-model-cache\") pod \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\" (UID: \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\") "
Nov 03 20:37:08.011682 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:08.011617    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-dshm\") pod \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\" (UID: \"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862\") "
Nov 03 20:37:08.011682 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:08.011646    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-tmp-dir" pod "7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" (UID: "7173cfa7-fb5e-40f3-9a2c-0f60ebef0862"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:08.012501 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:08.012192    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-model-cache" pod "7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" (UID: "7173cfa7-fb5e-40f3-9a2c-0f60ebef0862"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:08.012501 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:08.012393    5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-home\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:08.012501 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:08.012414    5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-tmp-dir\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:08.012501 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:08.012427    5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-model-cache\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:08.015101 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:08.015075    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-kube-api-access-7c4cp" pod "7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" (UID: "7173cfa7-fb5e-40f3-9a2c-0f60ebef0862"). InnerVolumeSpecName "kube-api-access-7c4cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:08.015195 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:08.015165    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-tls-certs" pod "7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" (UID: "7173cfa7-fb5e-40f3-9a2c-0f60ebef0862"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:37:08.015531 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:08.015511    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-dshm" pod "7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" (UID: "7173cfa7-fb5e-40f3-9a2c-0f60ebef0862"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:08.115195 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:08.114997    5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-tls-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:08.115195 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:08.115035    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c4cp\" (UniqueName: \"kubernetes.io/projected/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-kube-api-access-7c4cp\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:08.115195 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:08.115053    5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862-dshm\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:08.446450 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:08.446412    5591 generic.go:356] "Generic (PLEG): container finished" podID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerID="d03a7ea2a9efea2faa83bd80d81895629158b902359440342412831d2a55aaaf" exitCode=0
Nov 03 20:37:08.446663 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:08.446463    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"
Nov 03 20:37:08.446663 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:08.446501    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" event={"ID":"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862","Type":"ContainerDied","Data":"d03a7ea2a9efea2faa83bd80d81895629158b902359440342412831d2a55aaaf"}
Nov 03 20:37:08.446663 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:08.446544    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t" event={"ID":"7173cfa7-fb5e-40f3-9a2c-0f60ebef0862","Type":"ContainerDied","Data":"3bad33fc886df1cf3afc673c84900f3026f3c8bb8bcc001bd55cb67999db7731"}
Nov 03 20:37:08.446663 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:08.446562    5591 scope.go:122] "RemoveContainer" containerID="d03a7ea2a9efea2faa83bd80d81895629158b902359440342412831d2a55aaaf"
Nov 03 20:37:08.458932 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:08.458913    5591 scope.go:122] "RemoveContainer" containerID="d03a7ea2a9efea2faa83bd80d81895629158b902359440342412831d2a55aaaf"
Nov 03 20:37:08.459259 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:37:08.459233    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d03a7ea2a9efea2faa83bd80d81895629158b902359440342412831d2a55aaaf\": container with ID starting with d03a7ea2a9efea2faa83bd80d81895629158b902359440342412831d2a55aaaf not found: ID does not exist" containerID="d03a7ea2a9efea2faa83bd80d81895629158b902359440342412831d2a55aaaf"
Nov 03 20:37:08.459330 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:08.459268    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d03a7ea2a9efea2faa83bd80d81895629158b902359440342412831d2a55aaaf"} err="failed to get container status \"d03a7ea2a9efea2faa83bd80d81895629158b902359440342412831d2a55aaaf\": rpc error: code = NotFound desc = could not find container \"d03a7ea2a9efea2faa83bd80d81895629158b902359440342412831d2a55aaaf\": container with ID starting with d03a7ea2a9efea2faa83bd80d81895629158b902359440342412831d2a55aaaf not found: ID does not exist"
Nov 03 20:37:08.472688 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:08.472667    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"]
Nov 03 20:37:08.477780 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:08.477760    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d3d0df/distinct-2-w0-d3d0df-kserve-7fddc6fcd9-tvs8t"]
Nov 03 20:37:09.351584 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:09.351554    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" path="/var/lib/kubelet/pods/7173cfa7-fb5e-40f3-9a2c-0f60ebef0862/volumes"
Nov 03 20:37:11.565045 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:11.564953    5591 patch_prober.go:29] interesting pod/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:37:11.565045 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:11.565024    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm" podUID="e31559fd-2ec2-48cd-8f00-8167a567097b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:37:12.143426 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.143394    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 20:37:12.327804 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.315335    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e31559fd-2ec2-48cd-8f00-8167a567097b-kube-api-access-h9thn\" (UniqueName: \"kubernetes.io/projected/e31559fd-2ec2-48cd-8f00-8167a567097b-kube-api-access-h9thn\") pod \"e31559fd-2ec2-48cd-8f00-8167a567097b\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") "
Nov 03 20:37:12.327804 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.315696    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-workload-certs" pod "e31559fd-2ec2-48cd-8f00-8167a567097b" (UID: "e31559fd-2ec2-48cd-8f00-8167a567097b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:12.329532 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.329500    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-workload-certs\") pod \"e31559fd-2ec2-48cd-8f00-8167a567097b\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") "
Nov 03 20:37:12.329692 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.329676    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-token\" (UniqueName: \"kubernetes.io/projected/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-token\") pod \"e31559fd-2ec2-48cd-8f00-8167a567097b\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") "
Nov 03 20:37:12.331460 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.329780    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-data\") pod \"e31559fd-2ec2-48cd-8f00-8167a567097b\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") "
Nov 03 20:37:12.331460 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.329805    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-envoy\") pod \"e31559fd-2ec2-48cd-8f00-8167a567097b\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") "
Nov 03 20:37:12.331460 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.329828    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e31559fd-2ec2-48cd-8f00-8167a567097b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e31559fd-2ec2-48cd-8f00-8167a567097b-istiod-ca-cert\") pod \"e31559fd-2ec2-48cd-8f00-8167a567097b\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") "
Nov 03 20:37:12.331460 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.329872    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-podinfo\") pod \"e31559fd-2ec2-48cd-8f00-8167a567097b\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") "
Nov 03 20:37:12.331460 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.329934    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-workload-socket\") pod \"e31559fd-2ec2-48cd-8f00-8167a567097b\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") "
Nov 03 20:37:12.331460 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.329995    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-credential-socket\") pod \"e31559fd-2ec2-48cd-8f00-8167a567097b\" (UID: \"e31559fd-2ec2-48cd-8f00-8167a567097b\") "
Nov 03 20:37:12.331460 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.330566    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:12.331460 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.330816    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-credential-socket" pod "e31559fd-2ec2-48cd-8f00-8167a567097b" (UID: "e31559fd-2ec2-48cd-8f00-8167a567097b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:12.333962 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.333937    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-token" pod "e31559fd-2ec2-48cd-8f00-8167a567097b" (UID: "e31559fd-2ec2-48cd-8f00-8167a567097b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:12.334197 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.334167    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-data" pod "e31559fd-2ec2-48cd-8f00-8167a567097b" (UID: "e31559fd-2ec2-48cd-8f00-8167a567097b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:12.336734 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.336703    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-workload-socket" pod "e31559fd-2ec2-48cd-8f00-8167a567097b" (UID: "e31559fd-2ec2-48cd-8f00-8167a567097b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:12.336907 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.336884    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e31559fd-2ec2-48cd-8f00-8167a567097b-istiod-ca-cert" pod "e31559fd-2ec2-48cd-8f00-8167a567097b" (UID: "e31559fd-2ec2-48cd-8f00-8167a567097b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:12.338174 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.338151    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-envoy" pod "e31559fd-2ec2-48cd-8f00-8167a567097b" (UID: "e31559fd-2ec2-48cd-8f00-8167a567097b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:12.338516 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.338492    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-podinfo" pod "e31559fd-2ec2-48cd-8f00-8167a567097b" (UID: "e31559fd-2ec2-48cd-8f00-8167a567097b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:37:12.348463 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.348434    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e31559fd-2ec2-48cd-8f00-8167a567097b-kube-api-access-h9thn" pod "e31559fd-2ec2-48cd-8f00-8167a567097b" (UID: "e31559fd-2ec2-48cd-8f00-8167a567097b"). InnerVolumeSpecName "kube-api-access-h9thn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:12.431649 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.431612    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:12.431649 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.431640    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:12.431649 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.431650    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e31559fd-2ec2-48cd-8f00-8167a567097b-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:12.431953 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.431660    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:12.431953 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.431681    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:12.431953 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.431691    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e31559fd-2ec2-48cd-8f00-8167a567097b-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:12.431953 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.431700    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9thn\" (UniqueName: \"kubernetes.io/projected/e31559fd-2ec2-48cd-8f00-8167a567097b-kube-api-access-h9thn\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:12.431953 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.431714    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e31559fd-2ec2-48cd-8f00-8167a567097b-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:12.481578 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.481544    5591 generic.go:356] "Generic (PLEG): container finished" podID="e31559fd-2ec2-48cd-8f00-8167a567097b" containerID="0d4e4f3915c407f348d4baf6f79380faa66d002ca74f21e14d930c0248bd2501" exitCode=0
Nov 03 20:37:12.481801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.481605    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"
Nov 03 20:37:12.481801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.481640    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm" event={"ID":"e31559fd-2ec2-48cd-8f00-8167a567097b","Type":"ContainerDied","Data":"0d4e4f3915c407f348d4baf6f79380faa66d002ca74f21e14d930c0248bd2501"}
Nov 03 20:37:12.481801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.481701    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm" event={"ID":"e31559fd-2ec2-48cd-8f00-8167a567097b","Type":"ContainerDied","Data":"d5f314e297403c0645026e2ff759cef301903b54bfeddebc014e1761c4b9305e"}
Nov 03 20:37:12.481801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.481731    5591 scope.go:122] "RemoveContainer" containerID="0d4e4f3915c407f348d4baf6f79380faa66d002ca74f21e14d930c0248bd2501"
Nov 03 20:37:12.495132 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.495111    5591 scope.go:122] "RemoveContainer" containerID="0d4e4f3915c407f348d4baf6f79380faa66d002ca74f21e14d930c0248bd2501"
Nov 03 20:37:12.495468 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:37:12.495447    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d4e4f3915c407f348d4baf6f79380faa66d002ca74f21e14d930c0248bd2501\": container with ID starting with 0d4e4f3915c407f348d4baf6f79380faa66d002ca74f21e14d930c0248bd2501 not found: ID does not exist" containerID="0d4e4f3915c407f348d4baf6f79380faa66d002ca74f21e14d930c0248bd2501"
Nov 03 20:37:12.495517 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.495477    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d4e4f3915c407f348d4baf6f79380faa66d002ca74f21e14d930c0248bd2501"} err="failed to get container status \"0d4e4f3915c407f348d4baf6f79380faa66d002ca74f21e14d930c0248bd2501\": rpc error: code = NotFound desc = could not find container \"0d4e4f3915c407f348d4baf6f79380faa66d002ca74f21e14d930c0248bd2501\": container with ID starting with 0d4e4f3915c407f348d4baf6f79380faa66d002ca74f21e14d930c0248bd2501 not found: ID does not exist"
Nov 03 20:37:12.509010 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.508954    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"]
Nov 03 20:37:12.513376 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:12.513348    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3fbb1a-openshift-default-6f4dd6d84c-lg2sm"]
Nov 03 20:37:13.351947 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:13.351909    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e31559fd-2ec2-48cd-8f00-8167a567097b" path="/var/lib/kubelet/pods/e31559fd-2ec2-48cd-8f00-8167a567097b/volumes"
Nov 03 20:37:16.309932 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:16.309306    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4"]
Nov 03 20:37:16.309932 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:16.309555    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4" podUID="de96244c-7272-46a0-b20b-5cd1470fc8bc" containerName="maas-api" containerID="cri-o://788e1717bfc2ed8e2e903caec8f76a928a44183d343e3f142779c5a6713ed29b" gracePeriod=30
Nov 03 20:37:16.743877 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:16.743807    5591 generic.go:356] "Generic (PLEG): container finished" podID="de96244c-7272-46a0-b20b-5cd1470fc8bc" containerID="788e1717bfc2ed8e2e903caec8f76a928a44183d343e3f142779c5a6713ed29b" exitCode=0
Nov 03 20:37:16.744227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:16.744183    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4" event={"ID":"de96244c-7272-46a0-b20b-5cd1470fc8bc","Type":"ContainerDied","Data":"788e1717bfc2ed8e2e903caec8f76a928a44183d343e3f142779c5a6713ed29b"}
Nov 03 20:37:16.781396 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:16.781368    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4"
Nov 03 20:37:16.845292 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:16.845252    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de96244c-7272-46a0-b20b-5cd1470fc8bc-kube-api-access-jtrm5\" (UniqueName: \"kubernetes.io/projected/de96244c-7272-46a0-b20b-5cd1470fc8bc-kube-api-access-jtrm5\") pod \"de96244c-7272-46a0-b20b-5cd1470fc8bc\" (UID: \"de96244c-7272-46a0-b20b-5cd1470fc8bc\") "
Nov 03 20:37:16.845487 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:16.845371    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de96244c-7272-46a0-b20b-5cd1470fc8bc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de96244c-7272-46a0-b20b-5cd1470fc8bc-openshift-service-ca\") pod \"de96244c-7272-46a0-b20b-5cd1470fc8bc\" (UID: \"de96244c-7272-46a0-b20b-5cd1470fc8bc\") "
Nov 03 20:37:16.845563 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:16.845486    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de96244c-7272-46a0-b20b-5cd1470fc8bc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de96244c-7272-46a0-b20b-5cd1470fc8bc-metrics-certs\") pod \"de96244c-7272-46a0-b20b-5cd1470fc8bc\" (UID: \"de96244c-7272-46a0-b20b-5cd1470fc8bc\") "
Nov 03 20:37:16.845710 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:16.845695    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de96244c-7272-46a0-b20b-5cd1470fc8bc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de96244c-7272-46a0-b20b-5cd1470fc8bc-maas-api-tls\") pod \"de96244c-7272-46a0-b20b-5cd1470fc8bc\" (UID: \"de96244c-7272-46a0-b20b-5cd1470fc8bc\") "
Nov 03 20:37:16.845790 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:16.845739    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de96244c-7272-46a0-b20b-5cd1470fc8bc-openshift-service-ca" pod "de96244c-7272-46a0-b20b-5cd1470fc8bc" (UID: "de96244c-7272-46a0-b20b-5cd1470fc8bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:16.846350 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:16.846325    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de96244c-7272-46a0-b20b-5cd1470fc8bc-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:16.848037 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:16.848007    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de96244c-7272-46a0-b20b-5cd1470fc8bc-kube-api-access-jtrm5" pod "de96244c-7272-46a0-b20b-5cd1470fc8bc" (UID: "de96244c-7272-46a0-b20b-5cd1470fc8bc"). InnerVolumeSpecName "kube-api-access-jtrm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:16.848342 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:16.848315    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de96244c-7272-46a0-b20b-5cd1470fc8bc-metrics-certs" pod "de96244c-7272-46a0-b20b-5cd1470fc8bc" (UID: "de96244c-7272-46a0-b20b-5cd1470fc8bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:37:16.848455 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:16.848410    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de96244c-7272-46a0-b20b-5cd1470fc8bc-maas-api-tls" pod "de96244c-7272-46a0-b20b-5cd1470fc8bc" (UID: "de96244c-7272-46a0-b20b-5cd1470fc8bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:37:16.961232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:16.956064    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de96244c-7272-46a0-b20b-5cd1470fc8bc-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:16.961232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:16.956106    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtrm5\" (UniqueName: \"kubernetes.io/projected/de96244c-7272-46a0-b20b-5cd1470fc8bc-kube-api-access-jtrm5\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:16.961232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:16.956123    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de96244c-7272-46a0-b20b-5cd1470fc8bc-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:17.799613 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:17.799135    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4" event={"ID":"de96244c-7272-46a0-b20b-5cd1470fc8bc","Type":"ContainerDied","Data":"aa6783d61e97805283727f9bb532745f10b9db668408e5846f3b70a42db0185a"}
Nov 03 20:37:17.800387 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:17.800367    5591 scope.go:122] "RemoveContainer" containerID="788e1717bfc2ed8e2e903caec8f76a928a44183d343e3f142779c5a6713ed29b"
Nov 03 20:37:17.800656 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:17.800310    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4"
Nov 03 20:37:17.829096 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:17.829073    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4"]
Nov 03 20:37:17.833015 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:17.832992    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-d3d0df-7d96f997f7-5s4d4"]
Nov 03 20:37:19.351584 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:19.351549    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de96244c-7272-46a0-b20b-5cd1470fc8bc" path="/var/lib/kubelet/pods/de96244c-7272-46a0-b20b-5cd1470fc8bc/volumes"
Nov 03 20:37:21.740540 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.740503    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"]
Nov 03 20:37:21.741763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.741702    5591 cpu_manager.go:414] "containerMap: removing container" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main"
Nov 03 20:37:21.741763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.741730    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main"
Nov 03 20:37:21.741763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.741748    5591 cpu_manager.go:414] "containerMap: removing container" podUID="7aff0e54-0f19-41b1-af95-70aee78fea74" containerName="istio-proxy"
Nov 03 20:37:21.741763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.741756    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aff0e54-0f19-41b1-af95-70aee78fea74" containerName="istio-proxy"
Nov 03 20:37:21.742086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.741775    5591 cpu_manager.go:414] "containerMap: removing container" podUID="4a55d034-3d5e-41f1-ae14-b741d8b0c2a5" containerName="istio-proxy"
Nov 03 20:37:21.742086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.741784    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a55d034-3d5e-41f1-ae14-b741d8b0c2a5" containerName="istio-proxy"
Nov 03 20:37:21.742086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.741802    5591 cpu_manager.go:414] "containerMap: removing container" podUID="de96244c-7272-46a0-b20b-5cd1470fc8bc" containerName="maas-api"
Nov 03 20:37:21.742086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.741810    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de96244c-7272-46a0-b20b-5cd1470fc8bc" containerName="maas-api"
Nov 03 20:37:21.742086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.741823    5591 cpu_manager.go:414] "containerMap: removing container" podUID="a74714b9-b503-4817-b5d2-9ab74cd798a7" containerName="payload-processing"
Nov 03 20:37:21.742086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.741830    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74714b9-b503-4817-b5d2-9ab74cd798a7" containerName="payload-processing"
Nov 03 20:37:21.742086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.741853    5591 cpu_manager.go:414] "containerMap: removing container" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main"
Nov 03 20:37:21.742086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.741860    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main"
Nov 03 20:37:21.742086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.741875    5591 cpu_manager.go:414] "containerMap: removing container" podUID="d67ada42-276d-41e3-bb13-05a8bd3d1029" containerName="payload-pre-processing"
Nov 03 20:37:21.742086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.741882    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d67ada42-276d-41e3-bb13-05a8bd3d1029" containerName="payload-pre-processing"
Nov 03 20:37:21.742086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.741902    5591 cpu_manager.go:414] "containerMap: removing container" podUID="75752636-c0d6-46d6-b2d2-da4122853fac" containerName="payload-pre-processing"
Nov 03 20:37:21.742086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.741909    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75752636-c0d6-46d6-b2d2-da4122853fac" containerName="payload-pre-processing"
Nov 03 20:37:21.742086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.741923    5591 cpu_manager.go:414] "containerMap: removing container" podUID="2339afa1-517b-49cc-9d17-a037371e38e8" containerName="payload-processing"
Nov 03 20:37:21.742086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.741929    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2339afa1-517b-49cc-9d17-a037371e38e8" containerName="payload-processing"
Nov 03 20:37:21.742086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.741958    5591 cpu_manager.go:414] "containerMap: removing container" podUID="ed913799-a594-487c-92a1-f55b50cac03c" containerName="payload-pre-processing"
Nov 03 20:37:21.742086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.741965    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed913799-a594-487c-92a1-f55b50cac03c" containerName="payload-pre-processing"
Nov 03 20:37:21.742086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.741985    5591 cpu_manager.go:414] "containerMap: removing container" podUID="9a4ae1ca-060a-43e4-b877-02a2c3565899" containerName="maas-api"
Nov 03 20:37:21.742086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.741992    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a4ae1ca-060a-43e4-b877-02a2c3565899" containerName="maas-api"
Nov 03 20:37:21.742086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742023    5591 cpu_manager.go:414] "containerMap: removing container" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main"
Nov 03 20:37:21.742086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742029    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main"
Nov 03 20:37:21.742086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742041    5591 cpu_manager.go:414] "containerMap: removing container" podUID="ea3c9bb0-fa05-4815-8412-6db003469cd0" containerName="main"
Nov 03 20:37:21.742086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742048    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea3c9bb0-fa05-4815-8412-6db003469cd0" containerName="main"
Nov 03 20:37:21.742086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742061    5591 cpu_manager.go:414] "containerMap: removing container" podUID="0366c67a-080b-4983-8057-b0d9ef562477" containerName="maas-api"
Nov 03 20:37:21.742086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742068    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0366c67a-080b-4983-8057-b0d9ef562477" containerName="maas-api"
Nov 03 20:37:21.742086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742081    5591 cpu_manager.go:414] "containerMap: removing container" podUID="b83ce20e-9504-4cd1-864a-2bbafc41b804" containerName="revoke-keys"
Nov 03 20:37:21.742086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742088    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b83ce20e-9504-4cd1-864a-2bbafc41b804" containerName="revoke-keys"
Nov 03 20:37:21.742086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742102    5591 cpu_manager.go:414] "containerMap: removing container" podUID="150e0fa8-4c32-418d-ae65-384d1baf47b9" containerName="payload-processing"
Nov 03 20:37:21.744779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742111    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="150e0fa8-4c32-418d-ae65-384d1baf47b9" containerName="payload-processing"
Nov 03 20:37:21.744779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742123    5591 cpu_manager.go:414] "containerMap: removing container" podUID="e31559fd-2ec2-48cd-8f00-8167a567097b" containerName="istio-proxy"
Nov 03 20:37:21.744779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742130    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e31559fd-2ec2-48cd-8f00-8167a567097b" containerName="istio-proxy"
Nov 03 20:37:21.744779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742348    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="de96244c-7272-46a0-b20b-5cd1470fc8bc" containerName="maas-api"
Nov 03 20:37:21.744779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742369    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74714b9-b503-4817-b5d2-9ab74cd798a7" containerName="payload-processing"
Nov 03 20:37:21.744779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742386    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7173cfa7-fb5e-40f3-9a2c-0f60ebef0862" containerName="main"
Nov 03 20:37:21.744779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742399    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="d67ada42-276d-41e3-bb13-05a8bd3d1029" containerName="payload-pre-processing"
Nov 03 20:37:21.744779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742412    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="75752636-c0d6-46d6-b2d2-da4122853fac" containerName="payload-pre-processing"
Nov 03 20:37:21.744779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742422    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="2339afa1-517b-49cc-9d17-a037371e38e8" containerName="payload-processing"
Nov 03 20:37:21.744779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742435    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed913799-a594-487c-92a1-f55b50cac03c" containerName="payload-pre-processing"
Nov 03 20:37:21.744779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742445    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b83ce20e-9504-4cd1-864a-2bbafc41b804" containerName="revoke-keys"
Nov 03 20:37:21.744779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742453    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a4ae1ca-060a-43e4-b877-02a2c3565899" containerName="maas-api"
Nov 03 20:37:21.744779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742464    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c1b25f8-ea79-44e9-991f-1067e71cf960" containerName="main"
Nov 03 20:37:21.744779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742475    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea3c9bb0-fa05-4815-8412-6db003469cd0" containerName="main"
Nov 03 20:37:21.744779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742484    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0366c67a-080b-4983-8057-b0d9ef562477" containerName="maas-api"
Nov 03 20:37:21.744779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742494    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="150e0fa8-4c32-418d-ae65-384d1baf47b9" containerName="payload-processing"
Nov 03 20:37:21.744779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742502    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aff0e54-0f19-41b1-af95-70aee78fea74" containerName="istio-proxy"
Nov 03 20:37:21.744779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742510    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="e31559fd-2ec2-48cd-8f00-8167a567097b" containerName="istio-proxy"
Nov 03 20:37:21.744779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742521    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc613fc8-12f6-4cfb-9ab1-3a112eaced51" containerName="main"
Nov 03 20:37:21.744779 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.742533    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a55d034-3d5e-41f1-ae14-b741d8b0c2a5" containerName="istio-proxy"
Nov 03 20:37:21.748829 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.748798    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:21.757780 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.757740    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"]
Nov 03 20:37:21.891668 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.891634    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:21.891860 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.891812    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:21.891923 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.891860    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:21.891923 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.891883    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:21.892043 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.892018    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:21.892156 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.892135    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:21.892196 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.892178    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmf2d\" (UniqueName: \"kubernetes.io/projected/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-kube-api-access-pmf2d\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:21.892302 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.892285    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:21.892372 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:21.892358    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:22.008281 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:22.000404    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:22.013141 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:22.011751    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:22.013141 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:22.011853    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-data\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:22.013705 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:22.012407    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:22.013705 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:22.013117    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istiod-ca-cert\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:22.013705 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:22.013698    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:22.014045 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:22.013745    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:22.014045 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:22.013791    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:22.014045 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:22.013898    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:22.014045 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:22.013932    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmf2d\" (UniqueName: \"kubernetes.io/projected/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-kube-api-access-pmf2d\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:22.014045 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:22.013977    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:22.015119 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:22.015087    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-workload-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:22.015648 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:22.015614    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-credential-socket\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:22.019354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:22.015926    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-workload-certs\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:22.019354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:22.017722    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-podinfo\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:22.023075 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:22.023006    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-envoy\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:22.025248 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:22.025221    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-token\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:22.026831 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:22.026807    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmf2d\" (UniqueName: \"kubernetes.io/projected/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-kube-api-access-pmf2d\") pod \"e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") " pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:22.085133 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:22.085097    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:22.384006 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:22.383976    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"]
Nov 03 20:37:22.384732 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:37:22.384708    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81aa9ef2_7d1b_488c_bbd2_fba606a67f5b.slice/crio-4849e6ddc6d62b4733290c9ec674aa33f12f8ed2074d67991a5d1e984867c8dd WatchSource:0}: Error finding container 4849e6ddc6d62b4733290c9ec674aa33f12f8ed2074d67991a5d1e984867c8dd: Status 404 returned error can't find the container with id 4849e6ddc6d62b4733290c9ec674aa33f12f8ed2074d67991a5d1e984867c8dd
Nov 03 20:37:22.388276 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:22.388241    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:37:22.388374 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:22.388313    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:37:22.388374 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:22.388342    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:37:22.839711 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:22.839676    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq" event={"ID":"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b","Type":"ContainerStarted","Data":"65743d76152d7701e065534ffebaeced35df2392fd93c8b0303f5e48f78fcd96"}
Nov 03 20:37:22.840285 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:22.839729    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq" event={"ID":"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b","Type":"ContainerStarted","Data":"4849e6ddc6d62b4733290c9ec674aa33f12f8ed2074d67991a5d1e984867c8dd"}
Nov 03 20:37:22.862765 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:22.862716    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq" podStartSLOduration=1.862700742 podStartE2EDuration="1.862700742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:22.861565802 +0000 UTC m=+3591.633858821" watchObservedRunningTime="2026-11-03 20:37:22.862700742 +0000 UTC m=+3591.634993760"
Nov 03 20:37:23.091289 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:23.089573    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:23.091289 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:23.089746    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Nov 03 20:37:23.091289 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:23.089790    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq" podUID="81aa9ef2-7d1b-488c-bbd2-fba606a67f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Nov 03 20:37:24.101229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:24.092623    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Nov 03 20:37:24.101229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:24.092687    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq" podUID="81aa9ef2-7d1b-488c-bbd2-fba606a67f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Nov 03 20:37:25.102234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:25.101417    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Nov 03 20:37:25.102234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:25.101481    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq" podUID="81aa9ef2-7d1b-488c-bbd2-fba606a67f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Nov 03 20:37:26.091233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:26.090654    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Nov 03 20:37:26.091233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:26.090741    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq" podUID="81aa9ef2-7d1b-488c-bbd2-fba606a67f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Nov 03 20:37:27.101233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:27.096624    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Nov 03 20:37:27.101233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:27.096691    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq" podUID="81aa9ef2-7d1b-488c-bbd2-fba606a67f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Nov 03 20:37:28.094238 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:28.093649    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Nov 03 20:37:28.094238 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:28.093754    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq" podUID="81aa9ef2-7d1b-488c-bbd2-fba606a67f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Nov 03 20:37:28.732289 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:37:28.732252    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:37:29.086215 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:29.086178    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Nov 03 20:37:29.086403 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:29.086248    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq" podUID="81aa9ef2-7d1b-488c-bbd2-fba606a67f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Nov 03 20:37:30.091072 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:30.091040    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:30.091614 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:30.091594    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:30.092322 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:30.092298    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:32.639976 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:32.639933    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:37:34.101016 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:34.100985    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 20:37:34.102791 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:34.102544    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 20:37:40.021747 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:40.021109    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"]
Nov 03 20:37:40.021747 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:40.021700    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp" podUID="83260a88-c456-4334-81ec-f640ab303cc3" containerName="istio-proxy" containerID="cri-o://63b12d248fe3fede28096729d2efe8300f62ebe8f60c2125caf04ad205228c89" gracePeriod=30
Nov 03 20:37:40.031326 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:40.031284    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"]
Nov 03 20:37:40.031896 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:40.031866    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq" podUID="81aa9ef2-7d1b-488c-bbd2-fba606a67f5b" containerName="istio-proxy" containerID="cri-o://65743d76152d7701e065534ffebaeced35df2392fd93c8b0303f5e48f78fcd96" gracePeriod=30
Nov 03 20:37:40.457185 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:40.457101    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:37:40.457185 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:40.457173    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp" podUID="83260a88-c456-4334-81ec-f640ab303cc3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:37:45.093836 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.093801    5591 patch_prober.go:29] interesting pod/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body=
Nov 03 20:37:45.094312 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.093877    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq" podUID="81aa9ef2-7d1b-488c-bbd2-fba606a67f5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused"
Nov 03 20:37:45.353871 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.353843    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 20:37:45.435660 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.435389    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83260a88-c456-4334-81ec-f640ab303cc3-istio-token\" (UniqueName: \"kubernetes.io/projected/83260a88-c456-4334-81ec-f640ab303cc3-istio-token\") pod \"83260a88-c456-4334-81ec-f640ab303cc3\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") "
Nov 03 20:37:45.435660 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.435497    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-credential-socket\") pod \"83260a88-c456-4334-81ec-f640ab303cc3\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") "
Nov 03 20:37:45.435660 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.435558    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83260a88-c456-4334-81ec-f640ab303cc3-kube-api-access-jjpr8\" (UniqueName: \"kubernetes.io/projected/83260a88-c456-4334-81ec-f640ab303cc3-kube-api-access-jjpr8\") pod \"83260a88-c456-4334-81ec-f640ab303cc3\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") "
Nov 03 20:37:45.435660 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.435641    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-workload-certs\") pod \"83260a88-c456-4334-81ec-f640ab303cc3\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") "
Nov 03 20:37:45.436029 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.435717    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-istio-envoy\") pod \"83260a88-c456-4334-81ec-f640ab303cc3\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") "
Nov 03 20:37:45.436029 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.435777    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-workload-socket\") pod \"83260a88-c456-4334-81ec-f640ab303cc3\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") "
Nov 03 20:37:45.436029 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.435827    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-istio-data\") pod \"83260a88-c456-4334-81ec-f640ab303cc3\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") "
Nov 03 20:37:45.436029 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.435929    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83260a88-c456-4334-81ec-f640ab303cc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83260a88-c456-4334-81ec-f640ab303cc3-istio-podinfo\") pod \"83260a88-c456-4334-81ec-f640ab303cc3\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") "
Nov 03 20:37:45.436029 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.435999    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83260a88-c456-4334-81ec-f640ab303cc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83260a88-c456-4334-81ec-f640ab303cc3-istiod-ca-cert\") pod \"83260a88-c456-4334-81ec-f640ab303cc3\" (UID: \"83260a88-c456-4334-81ec-f640ab303cc3\") "
Nov 03 20:37:45.438469 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.438423    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83260a88-c456-4334-81ec-f640ab303cc3-istiod-ca-cert" pod "83260a88-c456-4334-81ec-f640ab303cc3" (UID: "83260a88-c456-4334-81ec-f640ab303cc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:45.438909 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.438880    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-istio-data" pod "83260a88-c456-4334-81ec-f640ab303cc3" (UID: "83260a88-c456-4334-81ec-f640ab303cc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:45.439348 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.439103    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-credential-socket" pod "83260a88-c456-4334-81ec-f640ab303cc3" (UID: "83260a88-c456-4334-81ec-f640ab303cc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:45.440708 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.440681    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-workload-socket" pod "83260a88-c456-4334-81ec-f640ab303cc3" (UID: "83260a88-c456-4334-81ec-f640ab303cc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:45.442684 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.442655    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-istio-envoy" pod "83260a88-c456-4334-81ec-f640ab303cc3" (UID: "83260a88-c456-4334-81ec-f640ab303cc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:45.442844 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.442828    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83260a88-c456-4334-81ec-f640ab303cc3-istio-token" pod "83260a88-c456-4334-81ec-f640ab303cc3" (UID: "83260a88-c456-4334-81ec-f640ab303cc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:45.442964 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.442948    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83260a88-c456-4334-81ec-f640ab303cc3-kube-api-access-jjpr8" pod "83260a88-c456-4334-81ec-f640ab303cc3" (UID: "83260a88-c456-4334-81ec-f640ab303cc3"). InnerVolumeSpecName "kube-api-access-jjpr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:45.443829 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.443806    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:45.446640 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.446612    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83260a88-c456-4334-81ec-f640ab303cc3-istio-podinfo" pod "83260a88-c456-4334-81ec-f640ab303cc3" (UID: "83260a88-c456-4334-81ec-f640ab303cc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:37:45.449404 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.449378    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-workload-certs" pod "83260a88-c456-4334-81ec-f640ab303cc3" (UID: "83260a88-c456-4334-81ec-f640ab303cc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:45.537740 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.537704    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-workload-socket\") pod \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") "
Nov 03 20:37:45.537961 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.537785    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-token\" (UniqueName: \"kubernetes.io/projected/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-token\") pod \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") "
Nov 03 20:37:45.537961 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.537818    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-podinfo\") pod \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") "
Nov 03 20:37:45.537961 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.537840    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-workload-certs\") pod \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") "
Nov 03 20:37:45.537961 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.537869    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-data\") pod \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") "
Nov 03 20:37:45.537961 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.537906    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istiod-ca-cert\") pod \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") "
Nov 03 20:37:45.538316 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.538023    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-envoy\") pod \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") "
Nov 03 20:37:45.538316 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.538027    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-workload-socket" pod "81aa9ef2-7d1b-488c-bbd2-fba606a67f5b" (UID: "81aa9ef2-7d1b-488c-bbd2-fba606a67f5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:45.538316 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.538048    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-credential-socket\") pod \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") "
Nov 03 20:37:45.538316 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.538186    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-kube-api-access-pmf2d\" (UniqueName: \"kubernetes.io/projected/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-kube-api-access-pmf2d\") pod \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\" (UID: \"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b\") "
Nov 03 20:37:45.538316 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.538308    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-workload-certs" pod "81aa9ef2-7d1b-488c-bbd2-fba606a67f5b" (UID: "81aa9ef2-7d1b-488c-bbd2-fba606a67f5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:45.538611 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.538415    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-credential-socket" pod "81aa9ef2-7d1b-488c-bbd2-fba606a67f5b" (UID: "81aa9ef2-7d1b-488c-bbd2-fba606a67f5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:45.538674 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.538657    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-data" pod "81aa9ef2-7d1b-488c-bbd2-fba606a67f5b" (UID: "81aa9ef2-7d1b-488c-bbd2-fba606a67f5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:45.538907 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.538884    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istiod-ca-cert" pod "81aa9ef2-7d1b-488c-bbd2-fba606a67f5b" (UID: "81aa9ef2-7d1b-488c-bbd2-fba606a67f5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:37:45.539287 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.539262    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:45.539390 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.539293    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:45.539390 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.539310    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:45.539390 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.539326    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:45.539390 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.539340    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:45.539390 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.539354    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:45.539390 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.539369    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83260a88-c456-4334-81ec-f640ab303cc3-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:45.539390 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.539383    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83260a88-c456-4334-81ec-f640ab303cc3-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:45.539962 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.539397    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:45.539962 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.539412    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83260a88-c456-4334-81ec-f640ab303cc3-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:45.539962 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.539426    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83260a88-c456-4334-81ec-f640ab303cc3-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:45.539962 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.539439    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjpr8\" (UniqueName: \"kubernetes.io/projected/83260a88-c456-4334-81ec-f640ab303cc3-kube-api-access-jjpr8\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:45.539962 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.539454    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:45.539962 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.539466    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:45.540548 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.540518    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-envoy" pod "81aa9ef2-7d1b-488c-bbd2-fba606a67f5b" (UID: "81aa9ef2-7d1b-488c-bbd2-fba606a67f5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:37:45.540645 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.540552    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-token" pod "81aa9ef2-7d1b-488c-bbd2-fba606a67f5b" (UID: "81aa9ef2-7d1b-488c-bbd2-fba606a67f5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:45.540687 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.540637    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-kube-api-access-pmf2d" pod "81aa9ef2-7d1b-488c-bbd2-fba606a67f5b" (UID: "81aa9ef2-7d1b-488c-bbd2-fba606a67f5b"). InnerVolumeSpecName "kube-api-access-pmf2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:37:45.541119 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.541102    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-podinfo" pod "81aa9ef2-7d1b-488c-bbd2-fba606a67f5b" (UID: "81aa9ef2-7d1b-488c-bbd2-fba606a67f5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:37:45.641367 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.641283    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:45.641367 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.641316    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmf2d\" (UniqueName: \"kubernetes.io/projected/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-kube-api-access-pmf2d\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:45.641367 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.641327    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:45.641367 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:45.641335    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:37:46.109381 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:46.109336    5591 generic.go:356] "Generic (PLEG): container finished" podID="81aa9ef2-7d1b-488c-bbd2-fba606a67f5b" containerID="65743d76152d7701e065534ffebaeced35df2392fd93c8b0303f5e48f78fcd96" exitCode=0
Nov 03 20:37:46.109801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:46.109399    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"
Nov 03 20:37:46.109801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:46.109426    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq" event={"ID":"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b","Type":"ContainerDied","Data":"65743d76152d7701e065534ffebaeced35df2392fd93c8b0303f5e48f78fcd96"}
Nov 03 20:37:46.109801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:46.109514    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq" event={"ID":"81aa9ef2-7d1b-488c-bbd2-fba606a67f5b","Type":"ContainerDied","Data":"4849e6ddc6d62b4733290c9ec674aa33f12f8ed2074d67991a5d1e984867c8dd"}
Nov 03 20:37:46.109801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:46.109537    5591 scope.go:122] "RemoveContainer" containerID="65743d76152d7701e065534ffebaeced35df2392fd93c8b0303f5e48f78fcd96"
Nov 03 20:37:46.111324 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:46.111297    5591 generic.go:356] "Generic (PLEG): container finished" podID="83260a88-c456-4334-81ec-f640ab303cc3" containerID="63b12d248fe3fede28096729d2efe8300f62ebe8f60c2125caf04ad205228c89" exitCode=0
Nov 03 20:37:46.111459 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:46.111330    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp" event={"ID":"83260a88-c456-4334-81ec-f640ab303cc3","Type":"ContainerDied","Data":"63b12d248fe3fede28096729d2efe8300f62ebe8f60c2125caf04ad205228c89"}
Nov 03 20:37:46.111459 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:46.111382    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp" event={"ID":"83260a88-c456-4334-81ec-f640ab303cc3","Type":"ContainerDied","Data":"395a6eef60cce290675629ab0af9f7c9e4dff70ff2dbbaff1b93224e02322fb8"}
Nov 03 20:37:46.111591 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:46.111388    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"
Nov 03 20:37:46.122609 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:46.122593    5591 scope.go:122] "RemoveContainer" containerID="65743d76152d7701e065534ffebaeced35df2392fd93c8b0303f5e48f78fcd96"
Nov 03 20:37:46.122905 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:37:46.122883    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65743d76152d7701e065534ffebaeced35df2392fd93c8b0303f5e48f78fcd96\": container with ID starting with 65743d76152d7701e065534ffebaeced35df2392fd93c8b0303f5e48f78fcd96 not found: ID does not exist" containerID="65743d76152d7701e065534ffebaeced35df2392fd93c8b0303f5e48f78fcd96"
Nov 03 20:37:46.122988 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:46.122916    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65743d76152d7701e065534ffebaeced35df2392fd93c8b0303f5e48f78fcd96"} err="failed to get container status \"65743d76152d7701e065534ffebaeced35df2392fd93c8b0303f5e48f78fcd96\": rpc error: code = NotFound desc = could not find container \"65743d76152d7701e065534ffebaeced35df2392fd93c8b0303f5e48f78fcd96\": container with ID starting with 65743d76152d7701e065534ffebaeced35df2392fd93c8b0303f5e48f78fcd96 not found: ID does not exist"
Nov 03 20:37:46.122988 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:46.122938    5591 scope.go:122] "RemoveContainer" containerID="63b12d248fe3fede28096729d2efe8300f62ebe8f60c2125caf04ad205228c89"
Nov 03 20:37:46.133901 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:46.133886    5591 scope.go:122] "RemoveContainer" containerID="63b12d248fe3fede28096729d2efe8300f62ebe8f60c2125caf04ad205228c89"
Nov 03 20:37:46.134155 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:37:46.134134    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63b12d248fe3fede28096729d2efe8300f62ebe8f60c2125caf04ad205228c89\": container with ID starting with 63b12d248fe3fede28096729d2efe8300f62ebe8f60c2125caf04ad205228c89 not found: ID does not exist" containerID="63b12d248fe3fede28096729d2efe8300f62ebe8f60c2125caf04ad205228c89"
Nov 03 20:37:46.134247 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:46.134165    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63b12d248fe3fede28096729d2efe8300f62ebe8f60c2125caf04ad205228c89"} err="failed to get container status \"63b12d248fe3fede28096729d2efe8300f62ebe8f60c2125caf04ad205228c89\": rpc error: code = NotFound desc = could not find container \"63b12d248fe3fede28096729d2efe8300f62ebe8f60c2125caf04ad205228c89\": container with ID starting with 63b12d248fe3fede28096729d2efe8300f62ebe8f60c2125caf04ad205228c89 not found: ID does not exist"
Nov 03 20:37:46.139415 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:46.139391    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"]
Nov 03 20:37:46.163223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:46.143887    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-mkbpq"]
Nov 03 20:37:46.181763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:46.181727    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"]
Nov 03 20:37:46.187618 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:46.187592    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d3d0df-openshift-default-7b6bc448b7-6lxfp"]
Nov 03 20:37:47.350621 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:47.350589    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81aa9ef2-7d1b-488c-bbd2-fba606a67f5b" path="/var/lib/kubelet/pods/81aa9ef2-7d1b-488c-bbd2-fba606a67f5b/volumes"
Nov 03 20:37:47.351137 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:37:47.351123    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83260a88-c456-4334-81ec-f640ab303cc3" path="/var/lib/kubelet/pods/83260a88-c456-4334-81ec-f640ab303cc3/volumes"
Nov 03 20:37:58.729428 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:37:58.729374    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:38:10.354304 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:10.354261    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:38:13.485055 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:13.485016    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"]
Nov 03 20:38:23.990732 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:23.990689    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"]
Nov 03 20:38:28.521298 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:28.521259    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw" podUID="143283f7-8182-41d6-bff0-e171c14a2836" containerName="main" containerID="cri-o://5ccb476c2ffca7b86d8124a7ea2f2dcc27eef205272b9291e3bf9357c65c3c34" gracePeriod=45
Nov 03 20:38:28.753451 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:38:28.753408    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:38:28.831375 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:28.831341    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"
Nov 03 20:38:28.918527 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:28.918491    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-tmp-dir\") pod \"143283f7-8182-41d6-bff0-e171c14a2836\" (UID: \"143283f7-8182-41d6-bff0-e171c14a2836\") "
Nov 03 20:38:28.918881 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:28.918852    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-dshm\" (UniqueName: \"kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-dshm\") pod \"143283f7-8182-41d6-bff0-e171c14a2836\" (UID: \"143283f7-8182-41d6-bff0-e171c14a2836\") "
Nov 03 20:38:28.919025 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:28.918924    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-model-cache\") pod \"143283f7-8182-41d6-bff0-e171c14a2836\" (UID: \"143283f7-8182-41d6-bff0-e171c14a2836\") "
Nov 03 20:38:28.919025 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:28.918959    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/143283f7-8182-41d6-bff0-e171c14a2836-kube-api-access-8w84j\" (UniqueName: \"kubernetes.io/projected/143283f7-8182-41d6-bff0-e171c14a2836-kube-api-access-8w84j\") pod \"143283f7-8182-41d6-bff0-e171c14a2836\" (UID: \"143283f7-8182-41d6-bff0-e171c14a2836\") "
Nov 03 20:38:28.919185 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:28.919116    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-home\" (UniqueName: \"kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-home\") pod \"143283f7-8182-41d6-bff0-e171c14a2836\" (UID: \"143283f7-8182-41d6-bff0-e171c14a2836\") "
Nov 03 20:38:28.919352 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:28.919331    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/143283f7-8182-41d6-bff0-e171c14a2836-tls-certs\" (UniqueName: \"kubernetes.io/secret/143283f7-8182-41d6-bff0-e171c14a2836-tls-certs\") pod \"143283f7-8182-41d6-bff0-e171c14a2836\" (UID: \"143283f7-8182-41d6-bff0-e171c14a2836\") "
Nov 03 20:38:28.919567 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:28.919535    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-model-cache" pod "143283f7-8182-41d6-bff0-e171c14a2836" (UID: "143283f7-8182-41d6-bff0-e171c14a2836"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:28.920005 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:28.919545    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-tmp-dir" pod "143283f7-8182-41d6-bff0-e171c14a2836" (UID: "143283f7-8182-41d6-bff0-e171c14a2836"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:28.920355 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:28.920314    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-home" pod "143283f7-8182-41d6-bff0-e171c14a2836" (UID: "143283f7-8182-41d6-bff0-e171c14a2836"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:28.920666 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:28.920645    5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-home\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:28.920795 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:28.920781    5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-tmp-dir\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:28.920903 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:28.920890    5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-model-cache\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:28.923343 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:28.923306    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/143283f7-8182-41d6-bff0-e171c14a2836-kube-api-access-8w84j" pod "143283f7-8182-41d6-bff0-e171c14a2836" (UID: "143283f7-8182-41d6-bff0-e171c14a2836"). InnerVolumeSpecName "kube-api-access-8w84j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:28.927755 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:28.927726    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-dshm" pod "143283f7-8182-41d6-bff0-e171c14a2836" (UID: "143283f7-8182-41d6-bff0-e171c14a2836"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:28.929064 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:28.929039    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/143283f7-8182-41d6-bff0-e171c14a2836-tls-certs" pod "143283f7-8182-41d6-bff0-e171c14a2836" (UID: "143283f7-8182-41d6-bff0-e171c14a2836"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:38:29.022272 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:29.022231    5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/143283f7-8182-41d6-bff0-e171c14a2836-tls-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:29.022272 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:29.022266    5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/143283f7-8182-41d6-bff0-e171c14a2836-dshm\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:29.022272 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:29.022277    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w84j\" (UniqueName: \"kubernetes.io/projected/143283f7-8182-41d6-bff0-e171c14a2836-kube-api-access-8w84j\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:29.532165 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:29.532117    5591 generic.go:356] "Generic (PLEG): container finished" podID="143283f7-8182-41d6-bff0-e171c14a2836" containerID="5ccb476c2ffca7b86d8124a7ea2f2dcc27eef205272b9291e3bf9357c65c3c34" exitCode=0
Nov 03 20:38:29.532165 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:29.532161    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"
Nov 03 20:38:29.532791 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:29.532186    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw" event={"ID":"143283f7-8182-41d6-bff0-e171c14a2836","Type":"ContainerDied","Data":"5ccb476c2ffca7b86d8124a7ea2f2dcc27eef205272b9291e3bf9357c65c3c34"}
Nov 03 20:38:29.532791 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:29.532275    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw" event={"ID":"143283f7-8182-41d6-bff0-e171c14a2836","Type":"ContainerDied","Data":"d14ca69abbeb21d7040fe74df4639052af4775d3162e344f722dd065c81e3924"}
Nov 03 20:38:29.532791 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:29.532299    5591 scope.go:122] "RemoveContainer" containerID="5ccb476c2ffca7b86d8124a7ea2f2dcc27eef205272b9291e3bf9357c65c3c34"
Nov 03 20:38:29.543926 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:29.543909    5591 scope.go:122] "RemoveContainer" containerID="5ccb476c2ffca7b86d8124a7ea2f2dcc27eef205272b9291e3bf9357c65c3c34"
Nov 03 20:38:29.544272 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:38:29.544248    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ccb476c2ffca7b86d8124a7ea2f2dcc27eef205272b9291e3bf9357c65c3c34\": container with ID starting with 5ccb476c2ffca7b86d8124a7ea2f2dcc27eef205272b9291e3bf9357c65c3c34 not found: ID does not exist" containerID="5ccb476c2ffca7b86d8124a7ea2f2dcc27eef205272b9291e3bf9357c65c3c34"
Nov 03 20:38:29.544369 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:29.544279    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ccb476c2ffca7b86d8124a7ea2f2dcc27eef205272b9291e3bf9357c65c3c34"} err="failed to get container status \"5ccb476c2ffca7b86d8124a7ea2f2dcc27eef205272b9291e3bf9357c65c3c34\": rpc error: code = NotFound desc = could not find container \"5ccb476c2ffca7b86d8124a7ea2f2dcc27eef205272b9291e3bf9357c65c3c34\": container with ID starting with 5ccb476c2ffca7b86d8124a7ea2f2dcc27eef205272b9291e3bf9357c65c3c34 not found: ID does not exist"
Nov 03 20:38:29.552110 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:29.552083    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"]
Nov 03 20:38:29.558510 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:29.558486    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-836265/rate-test-model-836265-kserve-bd9bbd87c-v4ppw"]
Nov 03 20:38:30.533020 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:30.532987    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc"]
Nov 03 20:38:31.352417 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:31.352384    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="143283f7-8182-41d6-bff0-e171c14a2836" path="/var/lib/kubelet/pods/143283f7-8182-41d6-bff0-e171c14a2836/volumes"
Nov 03 20:38:35.534282 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:35.534194    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc" podUID="b51ae7c4-0791-4250-809e-b558456a410b" containerName="payload-processing" containerID="cri-o://d101cb468d5e4c2a36a0875ad87b74fc242e2c92cc5ff901637169249d3f9e94" gracePeriod=25
Nov 03 20:38:35.917726 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:35.917693    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc"
Nov 03 20:38:35.953004 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:35.952970    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b51ae7c4-0791-4250-809e-b558456a410b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b51ae7c4-0791-4250-809e-b558456a410b-plugins-config-volume\") pod \"b51ae7c4-0791-4250-809e-b558456a410b\" (UID: \"b51ae7c4-0791-4250-809e-b558456a410b\") "
Nov 03 20:38:35.953250 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:35.953077    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b51ae7c4-0791-4250-809e-b558456a410b-kube-api-access-wgtmb\" (UniqueName: \"kubernetes.io/projected/b51ae7c4-0791-4250-809e-b558456a410b-kube-api-access-wgtmb\") pod \"b51ae7c4-0791-4250-809e-b558456a410b\" (UID: \"b51ae7c4-0791-4250-809e-b558456a410b\") "
Nov 03 20:38:35.953697 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:35.953664    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b51ae7c4-0791-4250-809e-b558456a410b-plugins-config-volume" pod "b51ae7c4-0791-4250-809e-b558456a410b" (UID: "b51ae7c4-0791-4250-809e-b558456a410b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:35.956772 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:35.956734    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b51ae7c4-0791-4250-809e-b558456a410b-kube-api-access-wgtmb" pod "b51ae7c4-0791-4250-809e-b558456a410b" (UID: "b51ae7c4-0791-4250-809e-b558456a410b"). InnerVolumeSpecName "kube-api-access-wgtmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:36.054692 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:36.054651    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b51ae7c4-0791-4250-809e-b558456a410b-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:36.054692 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:36.054687    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgtmb\" (UniqueName: \"kubernetes.io/projected/b51ae7c4-0791-4250-809e-b558456a410b-kube-api-access-wgtmb\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:36.631659 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:36.631623    5591 generic.go:356] "Generic (PLEG): container finished" podID="b51ae7c4-0791-4250-809e-b558456a410b" containerID="d101cb468d5e4c2a36a0875ad87b74fc242e2c92cc5ff901637169249d3f9e94" exitCode=0
Nov 03 20:38:36.632122 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:36.631679    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc"
Nov 03 20:38:36.632122 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:36.631705    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc" event={"ID":"b51ae7c4-0791-4250-809e-b558456a410b","Type":"ContainerDied","Data":"d101cb468d5e4c2a36a0875ad87b74fc242e2c92cc5ff901637169249d3f9e94"}
Nov 03 20:38:36.632122 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:36.631761    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc" event={"ID":"b51ae7c4-0791-4250-809e-b558456a410b","Type":"ContainerDied","Data":"e50822554f8b261ba8e82353afe34de9fad4c935915d9d803fc8b9ed1b3d24db"}
Nov 03 20:38:36.632122 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:36.631784    5591 scope.go:122] "RemoveContainer" containerID="d101cb468d5e4c2a36a0875ad87b74fc242e2c92cc5ff901637169249d3f9e94"
Nov 03 20:38:36.643648 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:36.643623    5591 scope.go:122] "RemoveContainer" containerID="d101cb468d5e4c2a36a0875ad87b74fc242e2c92cc5ff901637169249d3f9e94"
Nov 03 20:38:36.643978 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:38:36.643953    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d101cb468d5e4c2a36a0875ad87b74fc242e2c92cc5ff901637169249d3f9e94\": container with ID starting with d101cb468d5e4c2a36a0875ad87b74fc242e2c92cc5ff901637169249d3f9e94 not found: ID does not exist" containerID="d101cb468d5e4c2a36a0875ad87b74fc242e2c92cc5ff901637169249d3f9e94"
Nov 03 20:38:36.644074 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:36.643985    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d101cb468d5e4c2a36a0875ad87b74fc242e2c92cc5ff901637169249d3f9e94"} err="failed to get container status \"d101cb468d5e4c2a36a0875ad87b74fc242e2c92cc5ff901637169249d3f9e94\": rpc error: code = NotFound desc = could not find container \"d101cb468d5e4c2a36a0875ad87b74fc242e2c92cc5ff901637169249d3f9e94\": container with ID starting with d101cb468d5e4c2a36a0875ad87b74fc242e2c92cc5ff901637169249d3f9e94 not found: ID does not exist"
Nov 03 20:38:36.657054 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:36.657024    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc"]
Nov 03 20:38:36.660457 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:36.660431    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-eb69cb-578f87d45d-slgsc"]
Nov 03 20:38:37.351257 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:37.351218    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b51ae7c4-0791-4250-809e-b558456a410b" path="/var/lib/kubelet/pods/b51ae7c4-0791-4250-809e-b558456a410b/volumes"
Nov 03 20:38:39.023665 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.023595    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh" podUID="e10f31e1-5a95-4c6a-88ed-0d15291cb44e" containerName="main" containerID="cri-o://d51f6dfd474b025fbd9965f0367235b3acf497eacc0c4b619fb840bb4ff742da" gracePeriod=45
Nov 03 20:38:39.311966 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.311937    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"
Nov 03 20:38:39.417263 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.417229    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-tls-certs\" (UniqueName: \"kubernetes.io/secret/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-tls-certs\") pod \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\" (UID: \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\") "
Nov 03 20:38:39.417435 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.417288    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-tmp-dir\") pod \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\" (UID: \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\") "
Nov 03 20:38:39.417435 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.417405    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-dshm\") pod \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\" (UID: \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\") "
Nov 03 20:38:39.417538 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.417446    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-model-cache\") pod \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\" (UID: \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\") "
Nov 03 20:38:39.417538 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.417470    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-kube-api-access-4blwh\" (UniqueName: \"kubernetes.io/projected/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-kube-api-access-4blwh\") pod \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\" (UID: \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\") "
Nov 03 20:38:39.417631 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.417553    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-home\" (UniqueName: \"kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-home\") pod \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\" (UID: \"e10f31e1-5a95-4c6a-88ed-0d15291cb44e\") "
Nov 03 20:38:39.417631 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.417607    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-tmp-dir" pod "e10f31e1-5a95-4c6a-88ed-0d15291cb44e" (UID: "e10f31e1-5a95-4c6a-88ed-0d15291cb44e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:39.417904 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.417881    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-home" pod "e10f31e1-5a95-4c6a-88ed-0d15291cb44e" (UID: "e10f31e1-5a95-4c6a-88ed-0d15291cb44e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:39.418265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.418238    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-model-cache" pod "e10f31e1-5a95-4c6a-88ed-0d15291cb44e" (UID: "e10f31e1-5a95-4c6a-88ed-0d15291cb44e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:39.418513 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.418491    5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-model-cache\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:39.418608 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.418521    5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-home\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:39.418608 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.418538    5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-tmp-dir\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:39.419857 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.419835    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-dshm" pod "e10f31e1-5a95-4c6a-88ed-0d15291cb44e" (UID: "e10f31e1-5a95-4c6a-88ed-0d15291cb44e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:39.419957 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.419908    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-tls-certs" pod "e10f31e1-5a95-4c6a-88ed-0d15291cb44e" (UID: "e10f31e1-5a95-4c6a-88ed-0d15291cb44e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:38:39.419957 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.419918    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-kube-api-access-4blwh" pod "e10f31e1-5a95-4c6a-88ed-0d15291cb44e" (UID: "e10f31e1-5a95-4c6a-88ed-0d15291cb44e"). InnerVolumeSpecName "kube-api-access-4blwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:39.520417 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.520384    5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-tls-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:39.520417 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.520412    5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-dshm\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:39.520417 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.520422    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4blwh\" (UniqueName: \"kubernetes.io/projected/e10f31e1-5a95-4c6a-88ed-0d15291cb44e-kube-api-access-4blwh\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:39.655916 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.655830    5591 generic.go:356] "Generic (PLEG): container finished" podID="e10f31e1-5a95-4c6a-88ed-0d15291cb44e" containerID="d51f6dfd474b025fbd9965f0367235b3acf497eacc0c4b619fb840bb4ff742da" exitCode=0
Nov 03 20:38:39.655916 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.655874    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"
Nov 03 20:38:39.656138 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.655920    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh" event={"ID":"e10f31e1-5a95-4c6a-88ed-0d15291cb44e","Type":"ContainerDied","Data":"d51f6dfd474b025fbd9965f0367235b3acf497eacc0c4b619fb840bb4ff742da"}
Nov 03 20:38:39.656138 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.655960    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh" event={"ID":"e10f31e1-5a95-4c6a-88ed-0d15291cb44e","Type":"ContainerDied","Data":"d62049e23135f2ce8da8ebf9a4da46f4015185e29477da36bd0f5d89da56aa30"}
Nov 03 20:38:39.656138 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.655979    5591 scope.go:122] "RemoveContainer" containerID="d51f6dfd474b025fbd9965f0367235b3acf497eacc0c4b619fb840bb4ff742da"
Nov 03 20:38:39.668173 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.668152    5591 scope.go:122] "RemoveContainer" containerID="d51f6dfd474b025fbd9965f0367235b3acf497eacc0c4b619fb840bb4ff742da"
Nov 03 20:38:39.668544 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:38:39.668518    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d51f6dfd474b025fbd9965f0367235b3acf497eacc0c4b619fb840bb4ff742da\": container with ID starting with d51f6dfd474b025fbd9965f0367235b3acf497eacc0c4b619fb840bb4ff742da not found: ID does not exist" containerID="d51f6dfd474b025fbd9965f0367235b3acf497eacc0c4b619fb840bb4ff742da"
Nov 03 20:38:39.668631 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.668553    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d51f6dfd474b025fbd9965f0367235b3acf497eacc0c4b619fb840bb4ff742da"} err="failed to get container status \"d51f6dfd474b025fbd9965f0367235b3acf497eacc0c4b619fb840bb4ff742da\": rpc error: code = NotFound desc = could not find container \"d51f6dfd474b025fbd9965f0367235b3acf497eacc0c4b619fb840bb4ff742da\": container with ID starting with d51f6dfd474b025fbd9965f0367235b3acf497eacc0c4b619fb840bb4ff742da not found: ID does not exist"
Nov 03 20:38:39.680907 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.680880    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"]
Nov 03 20:38:39.686343 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:39.686317    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-660d95/rate-test-model-660d95-kserve-65df4f7c57-xj8jh"]
Nov 03 20:38:41.352066 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:41.352031    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e10f31e1-5a95-4c6a-88ed-0d15291cb44e" path="/var/lib/kubelet/pods/e10f31e1-5a95-4c6a-88ed-0d15291cb44e/volumes"
Nov 03 20:38:48.129905 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:48.129869    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"]
Nov 03 20:38:48.130398 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:48.130128    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg" podUID="824d78ad-b227-4bdd-8245-6f37ef395666" containerName="istio-proxy" containerID="cri-o://fd411a736a6c9774b699a67fff682aa01b3a17a335e7209c49ea50f12376341b" gracePeriod=30
Nov 03 20:38:48.134187 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:48.134159    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"]
Nov 03 20:38:48.134508 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:48.134397    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt" podUID="120358c4-5509-4cc4-8522-647ddbde2d90" containerName="istio-proxy" containerID="cri-o://07f19dbfe7cf353f847f514c30cb11bb1df4e5a2501fe576cab388803dddee5d" gracePeriod=30
Nov 03 20:38:51.461302 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:51.461244    5591 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:38:51.461714 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:51.461335    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg" podUID="824d78ad-b227-4bdd-8245-6f37ef395666" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:38:51.505759 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:51.505720    5591 patch_prober.go:29] interesting pod/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:38:51.505960 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:51.505790    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt" podUID="120358c4-5509-4cc4-8522-647ddbde2d90" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:38:53.525973 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.525458    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:38:53.527957 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.527934    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:38:53.604193 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.604150    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824d78ad-b227-4bdd-8245-6f37ef395666-kube-api-access-rsj54\" (UniqueName: \"kubernetes.io/projected/824d78ad-b227-4bdd-8245-6f37ef395666-kube-api-access-rsj54\") pod \"824d78ad-b227-4bdd-8245-6f37ef395666\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") "
Nov 03 20:38:53.604427 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.604367    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/120358c4-5509-4cc4-8522-647ddbde2d90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/120358c4-5509-4cc4-8522-647ddbde2d90-istiod-ca-cert\") pod \"120358c4-5509-4cc4-8522-647ddbde2d90\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") "
Nov 03 20:38:53.604427 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.604403    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-credential-socket\") pod \"120358c4-5509-4cc4-8522-647ddbde2d90\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") "
Nov 03 20:38:53.604544 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.604491    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-credential-socket\") pod \"824d78ad-b227-4bdd-8245-6f37ef395666\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") "
Nov 03 20:38:53.604544 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.604530    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/120358c4-5509-4cc4-8522-647ddbde2d90-istio-token\" (UniqueName: \"kubernetes.io/projected/120358c4-5509-4cc4-8522-647ddbde2d90-istio-token\") pod \"120358c4-5509-4cc4-8522-647ddbde2d90\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") "
Nov 03 20:38:53.604657 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.604579    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-istio-data\") pod \"824d78ad-b227-4bdd-8245-6f37ef395666\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") "
Nov 03 20:38:53.604657 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.604613    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-workload-socket\") pod \"120358c4-5509-4cc4-8522-647ddbde2d90\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") "
Nov 03 20:38:53.604657 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.604642    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/120358c4-5509-4cc4-8522-647ddbde2d90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/120358c4-5509-4cc4-8522-647ddbde2d90-istio-podinfo\") pod \"120358c4-5509-4cc4-8522-647ddbde2d90\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") "
Nov 03 20:38:53.604823 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.604773    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-istio-envoy\") pod \"120358c4-5509-4cc4-8522-647ddbde2d90\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") "
Nov 03 20:38:53.604823 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.604783    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/120358c4-5509-4cc4-8522-647ddbde2d90-istiod-ca-cert" pod "120358c4-5509-4cc4-8522-647ddbde2d90" (UID: "120358c4-5509-4cc4-8522-647ddbde2d90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:53.604823 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.604814    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/824d78ad-b227-4bdd-8245-6f37ef395666-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/824d78ad-b227-4bdd-8245-6f37ef395666-istiod-ca-cert\") pod \"824d78ad-b227-4bdd-8245-6f37ef395666\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") "
Nov 03 20:38:53.604986 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.604870    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/120358c4-5509-4cc4-8522-647ddbde2d90-kube-api-access-pvdtp\" (UniqueName: \"kubernetes.io/projected/120358c4-5509-4cc4-8522-647ddbde2d90-kube-api-access-pvdtp\") pod \"120358c4-5509-4cc4-8522-647ddbde2d90\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") "
Nov 03 20:38:53.604986 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.604943    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-workload-socket\") pod \"824d78ad-b227-4bdd-8245-6f37ef395666\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") "
Nov 03 20:38:53.605102 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.605004    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-workload-certs\") pod \"120358c4-5509-4cc4-8522-647ddbde2d90\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") "
Nov 03 20:38:53.605102 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.605043    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824d78ad-b227-4bdd-8245-6f37ef395666-istio-token\" (UniqueName: \"kubernetes.io/projected/824d78ad-b227-4bdd-8245-6f37ef395666-istio-token\") pod \"824d78ad-b227-4bdd-8245-6f37ef395666\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") "
Nov 03 20:38:53.605102 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.605081    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-workload-socket" pod "120358c4-5509-4cc4-8522-647ddbde2d90" (UID: "120358c4-5509-4cc4-8522-647ddbde2d90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:53.605289 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.605136    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/824d78ad-b227-4bdd-8245-6f37ef395666-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/824d78ad-b227-4bdd-8245-6f37ef395666-istio-podinfo\") pod \"824d78ad-b227-4bdd-8245-6f37ef395666\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") "
Nov 03 20:38:53.605350 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.605299    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-istio-data" pod "824d78ad-b227-4bdd-8245-6f37ef395666" (UID: "824d78ad-b227-4bdd-8245-6f37ef395666"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:53.605500 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.605480    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-workload-certs" pod "120358c4-5509-4cc4-8522-647ddbde2d90" (UID: "120358c4-5509-4cc4-8522-647ddbde2d90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:53.605740 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.605718    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-workload-socket" pod "824d78ad-b227-4bdd-8245-6f37ef395666" (UID: "824d78ad-b227-4bdd-8245-6f37ef395666"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:53.607354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.605907    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-credential-socket" pod "824d78ad-b227-4bdd-8245-6f37ef395666" (UID: "824d78ad-b227-4bdd-8245-6f37ef395666"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:53.607354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.607170    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/120358c4-5509-4cc4-8522-647ddbde2d90-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:53.607354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.607191    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:53.607354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.607244    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:53.607354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.607259    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:53.607354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.607272    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:53.607354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.607284    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:53.608279 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.608253    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-credential-socket" pod "120358c4-5509-4cc4-8522-647ddbde2d90" (UID: "120358c4-5509-4cc4-8522-647ddbde2d90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:53.608437 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.608311    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/824d78ad-b227-4bdd-8245-6f37ef395666-istiod-ca-cert" pod "824d78ad-b227-4bdd-8245-6f37ef395666" (UID: "824d78ad-b227-4bdd-8245-6f37ef395666"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:38:53.609998 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.609973    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/120358c4-5509-4cc4-8522-647ddbde2d90-istio-podinfo" pod "120358c4-5509-4cc4-8522-647ddbde2d90" (UID: "120358c4-5509-4cc4-8522-647ddbde2d90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:38:53.610148 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.610103    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/120358c4-5509-4cc4-8522-647ddbde2d90-kube-api-access-pvdtp" pod "120358c4-5509-4cc4-8522-647ddbde2d90" (UID: "120358c4-5509-4cc4-8522-647ddbde2d90"). InnerVolumeSpecName "kube-api-access-pvdtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:53.612655 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.612628    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/824d78ad-b227-4bdd-8245-6f37ef395666-istio-podinfo" pod "824d78ad-b227-4bdd-8245-6f37ef395666" (UID: "824d78ad-b227-4bdd-8245-6f37ef395666"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:38:53.617016 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.616984    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/120358c4-5509-4cc4-8522-647ddbde2d90-istio-token" pod "120358c4-5509-4cc4-8522-647ddbde2d90" (UID: "120358c4-5509-4cc4-8522-647ddbde2d90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:53.617119 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.617027    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824d78ad-b227-4bdd-8245-6f37ef395666-istio-token" pod "824d78ad-b227-4bdd-8245-6f37ef395666" (UID: "824d78ad-b227-4bdd-8245-6f37ef395666"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:53.617192 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.617168    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824d78ad-b227-4bdd-8245-6f37ef395666-kube-api-access-rsj54" pod "824d78ad-b227-4bdd-8245-6f37ef395666" (UID: "824d78ad-b227-4bdd-8245-6f37ef395666"). InnerVolumeSpecName "kube-api-access-rsj54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:38:53.618395 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.618371    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-istio-envoy" pod "120358c4-5509-4cc4-8522-647ddbde2d90" (UID: "120358c4-5509-4cc4-8522-647ddbde2d90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:53.708246 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.708138    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-istio-envoy\") pod \"824d78ad-b227-4bdd-8245-6f37ef395666\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") "
Nov 03 20:38:53.708452 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.708433    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-istio-data\") pod \"120358c4-5509-4cc4-8522-647ddbde2d90\" (UID: \"120358c4-5509-4cc4-8522-647ddbde2d90\") "
Nov 03 20:38:53.708522 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.708459    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-workload-certs\") pod \"824d78ad-b227-4bdd-8245-6f37ef395666\" (UID: \"824d78ad-b227-4bdd-8245-6f37ef395666\") "
Nov 03 20:38:53.708861 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.708833    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-workload-certs" pod "824d78ad-b227-4bdd-8245-6f37ef395666" (UID: "824d78ad-b227-4bdd-8245-6f37ef395666"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:53.709015 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.708997    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsj54\" (UniqueName: \"kubernetes.io/projected/824d78ad-b227-4bdd-8245-6f37ef395666-kube-api-access-rsj54\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:53.709093 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.709022    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:53.709093 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.709037    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/120358c4-5509-4cc4-8522-647ddbde2d90-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:53.709093 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.709050    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/120358c4-5509-4cc4-8522-647ddbde2d90-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:53.709093 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.709064    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:53.709093 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.709078    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/824d78ad-b227-4bdd-8245-6f37ef395666-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:53.709313 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.709092    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvdtp\" (UniqueName: \"kubernetes.io/projected/120358c4-5509-4cc4-8522-647ddbde2d90-kube-api-access-pvdtp\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:53.709313 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.709106    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/824d78ad-b227-4bdd-8245-6f37ef395666-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:53.709313 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.709122    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/824d78ad-b227-4bdd-8245-6f37ef395666-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:53.709313 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.709136    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:53.709313 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.708751    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-istio-data" pod "120358c4-5509-4cc4-8522-647ddbde2d90" (UID: "120358c4-5509-4cc4-8522-647ddbde2d90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:53.710595 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.710570    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-istio-envoy" pod "824d78ad-b227-4bdd-8245-6f37ef395666" (UID: "824d78ad-b227-4bdd-8245-6f37ef395666"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:38:53.810958 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.810924    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/120358c4-5509-4cc4-8522-647ddbde2d90-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:53.810958 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.810952    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/824d78ad-b227-4bdd-8245-6f37ef395666-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:38:53.811450 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.811424    5591 generic.go:356] "Generic (PLEG): container finished" podID="120358c4-5509-4cc4-8522-647ddbde2d90" containerID="07f19dbfe7cf353f847f514c30cb11bb1df4e5a2501fe576cab388803dddee5d" exitCode=0
Nov 03 20:38:53.811543 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.811473    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"
Nov 03 20:38:53.811543 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.811510    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt" event={"ID":"120358c4-5509-4cc4-8522-647ddbde2d90","Type":"ContainerDied","Data":"07f19dbfe7cf353f847f514c30cb11bb1df4e5a2501fe576cab388803dddee5d"}
Nov 03 20:38:53.811645 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.811564    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt" event={"ID":"120358c4-5509-4cc4-8522-647ddbde2d90","Type":"ContainerDied","Data":"ff5d2971dd84a8e0d9ac1b56aaa796f89a56d707bfa7172d441ae3305bcf5f53"}
Nov 03 20:38:53.811645 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.811594    5591 scope.go:122] "RemoveContainer" containerID="07f19dbfe7cf353f847f514c30cb11bb1df4e5a2501fe576cab388803dddee5d"
Nov 03 20:38:53.813270 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.813196    5591 generic.go:356] "Generic (PLEG): container finished" podID="824d78ad-b227-4bdd-8245-6f37ef395666" containerID="fd411a736a6c9774b699a67fff682aa01b3a17a335e7209c49ea50f12376341b" exitCode=0
Nov 03 20:38:53.813408 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.813329    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg" event={"ID":"824d78ad-b227-4bdd-8245-6f37ef395666","Type":"ContainerDied","Data":"fd411a736a6c9774b699a67fff682aa01b3a17a335e7209c49ea50f12376341b"}
Nov 03 20:38:53.813408 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.813374    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg" event={"ID":"824d78ad-b227-4bdd-8245-6f37ef395666","Type":"ContainerDied","Data":"90ed241708fdd35a6a170a80b906648cf0bc89c7fc432db72e0650f17f232afd"}
Nov 03 20:38:53.813577 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.813443    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"
Nov 03 20:38:53.831105 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.831080    5591 scope.go:122] "RemoveContainer" containerID="07f19dbfe7cf353f847f514c30cb11bb1df4e5a2501fe576cab388803dddee5d"
Nov 03 20:38:53.832364 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:38:53.831595    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07f19dbfe7cf353f847f514c30cb11bb1df4e5a2501fe576cab388803dddee5d\": container with ID starting with 07f19dbfe7cf353f847f514c30cb11bb1df4e5a2501fe576cab388803dddee5d not found: ID does not exist" containerID="07f19dbfe7cf353f847f514c30cb11bb1df4e5a2501fe576cab388803dddee5d"
Nov 03 20:38:53.832364 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.831632    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07f19dbfe7cf353f847f514c30cb11bb1df4e5a2501fe576cab388803dddee5d"} err="failed to get container status \"07f19dbfe7cf353f847f514c30cb11bb1df4e5a2501fe576cab388803dddee5d\": rpc error: code = NotFound desc = could not find container \"07f19dbfe7cf353f847f514c30cb11bb1df4e5a2501fe576cab388803dddee5d\": container with ID starting with 07f19dbfe7cf353f847f514c30cb11bb1df4e5a2501fe576cab388803dddee5d not found: ID does not exist"
Nov 03 20:38:53.832364 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.831654    5591 scope.go:122] "RemoveContainer" containerID="fd411a736a6c9774b699a67fff682aa01b3a17a335e7209c49ea50f12376341b"
Nov 03 20:38:53.846984 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.846952    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"]
Nov 03 20:38:53.848864 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.848832    5591 scope.go:122] "RemoveContainer" containerID="fd411a736a6c9774b699a67fff682aa01b3a17a335e7209c49ea50f12376341b"
Nov 03 20:38:53.849353 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:38:53.849319    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd411a736a6c9774b699a67fff682aa01b3a17a335e7209c49ea50f12376341b\": container with ID starting with fd411a736a6c9774b699a67fff682aa01b3a17a335e7209c49ea50f12376341b not found: ID does not exist" containerID="fd411a736a6c9774b699a67fff682aa01b3a17a335e7209c49ea50f12376341b"
Nov 03 20:38:53.849447 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.849351    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd411a736a6c9774b699a67fff682aa01b3a17a335e7209c49ea50f12376341b"} err="failed to get container status \"fd411a736a6c9774b699a67fff682aa01b3a17a335e7209c49ea50f12376341b\": rpc error: code = NotFound desc = could not find container \"fd411a736a6c9774b699a67fff682aa01b3a17a335e7209c49ea50f12376341b\": container with ID starting with fd411a736a6c9774b699a67fff682aa01b3a17a335e7209c49ea50f12376341b not found: ID does not exist"
Nov 03 20:38:53.852270 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.852189    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-5lvpg"]
Nov 03 20:38:53.867424 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.867384    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"]
Nov 03 20:38:53.871685 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:53.871651    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-eb69cb-openshift-default-5f5b7678f7-vplgt"]
Nov 03 20:38:55.355384 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:55.355341    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="120358c4-5509-4cc4-8522-647ddbde2d90" path="/var/lib/kubelet/pods/120358c4-5509-4cc4-8522-647ddbde2d90/volumes"
Nov 03 20:38:55.356176 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:55.356153    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="824d78ad-b227-4bdd-8245-6f37ef395666" path="/var/lib/kubelet/pods/824d78ad-b227-4bdd-8245-6f37ef395666/volumes"
Nov 03 20:38:55.822456 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:38:55.822418    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:38:58.734379 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:38:58.734342    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:39:04.802858 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:04.802815    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"]
Nov 03 20:39:04.803474 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:04.803096    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67" podUID="a884ccd6-a398-44d2-a3a7-40c8e96abd66" containerName="istio-proxy" containerID="cri-o://c7b491a1f94b7540bb8bd0aba6f4e2823ae4521ea8a70fd0f3d86374d9c6330b" gracePeriod=30
Nov 03 20:39:06.936037 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:06.935998    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:39:06.936574 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:06.936071    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67" podUID="a884ccd6-a398-44d2-a3a7-40c8e96abd66" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:39:10.020637 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.020598    5591 generic.go:356] "Generic (PLEG): container finished" podID="a884ccd6-a398-44d2-a3a7-40c8e96abd66" containerID="c7b491a1f94b7540bb8bd0aba6f4e2823ae4521ea8a70fd0f3d86374d9c6330b" exitCode=0
Nov 03 20:39:10.021035 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.020665    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67" event={"ID":"a884ccd6-a398-44d2-a3a7-40c8e96abd66","Type":"ContainerDied","Data":"c7b491a1f94b7540bb8bd0aba6f4e2823ae4521ea8a70fd0f3d86374d9c6330b"}
Nov 03 20:39:10.176939 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.176911    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:39:10.237073 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.237037    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-workload-socket\") pod \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") "
Nov 03 20:39:10.237073 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.237088    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-envoy\") pod \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") "
Nov 03 20:39:10.237405 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.237112    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-token\" (UniqueName: \"kubernetes.io/projected/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-token\") pod \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") "
Nov 03 20:39:10.237405 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.237187    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a884ccd6-a398-44d2-a3a7-40c8e96abd66-kube-api-access-pcvx6\" (UniqueName: \"kubernetes.io/projected/a884ccd6-a398-44d2-a3a7-40c8e96abd66-kube-api-access-pcvx6\") pod \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") "
Nov 03 20:39:10.237547 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.237416    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-workload-socket" pod "a884ccd6-a398-44d2-a3a7-40c8e96abd66" (UID: "a884ccd6-a398-44d2-a3a7-40c8e96abd66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:10.237547 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.237423    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-workload-certs\") pod \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") "
Nov 03 20:39:10.237547 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.237502    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-data\") pod \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") "
Nov 03 20:39:10.237723 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.237623    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-workload-certs" pod "a884ccd6-a398-44d2-a3a7-40c8e96abd66" (UID: "a884ccd6-a398-44d2-a3a7-40c8e96abd66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:10.237723 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.237652    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-podinfo\") pod \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") "
Nov 03 20:39:10.237847 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.237740    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-credential-socket\") pod \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") "
Nov 03 20:39:10.237847 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.237779    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istiod-ca-cert\") pod \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\" (UID: \"a884ccd6-a398-44d2-a3a7-40c8e96abd66\") "
Nov 03 20:39:10.237847 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.237822    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-data" pod "a884ccd6-a398-44d2-a3a7-40c8e96abd66" (UID: "a884ccd6-a398-44d2-a3a7-40c8e96abd66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:10.238316 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.238287    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-credential-socket" pod "a884ccd6-a398-44d2-a3a7-40c8e96abd66" (UID: "a884ccd6-a398-44d2-a3a7-40c8e96abd66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:10.238866 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.238835    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:10.238996 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.238872    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:10.238996 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.238890    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:10.238996 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.238903    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:10.239185 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.239072    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istiod-ca-cert" pod "a884ccd6-a398-44d2-a3a7-40c8e96abd66" (UID: "a884ccd6-a398-44d2-a3a7-40c8e96abd66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:10.239940 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.239808    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-envoy" pod "a884ccd6-a398-44d2-a3a7-40c8e96abd66" (UID: "a884ccd6-a398-44d2-a3a7-40c8e96abd66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:10.242600 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.242571    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-podinfo" pod "a884ccd6-a398-44d2-a3a7-40c8e96abd66" (UID: "a884ccd6-a398-44d2-a3a7-40c8e96abd66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:39:10.242710 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.242612    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a884ccd6-a398-44d2-a3a7-40c8e96abd66-kube-api-access-pcvx6" pod "a884ccd6-a398-44d2-a3a7-40c8e96abd66" (UID: "a884ccd6-a398-44d2-a3a7-40c8e96abd66"). InnerVolumeSpecName "kube-api-access-pcvx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:10.244536 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.244507    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-token" pod "a884ccd6-a398-44d2-a3a7-40c8e96abd66" (UID: "a884ccd6-a398-44d2-a3a7-40c8e96abd66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:10.340676 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.340638    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:10.340676 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.340681    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:10.340895 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.340691    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:10.340895 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.340701    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a884ccd6-a398-44d2-a3a7-40c8e96abd66-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:10.340895 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:10.340710    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcvx6\" (UniqueName: \"kubernetes.io/projected/a884ccd6-a398-44d2-a3a7-40c8e96abd66-kube-api-access-pcvx6\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:11.030556 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:11.030523    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"
Nov 03 20:39:11.031023 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:11.030553    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67" event={"ID":"a884ccd6-a398-44d2-a3a7-40c8e96abd66","Type":"ContainerDied","Data":"a964cf41c0481b9a7dfe6f8d05609563c1cdcbc0658c4993382dfb87295accc5"}
Nov 03 20:39:11.031023 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:11.030614    5591 scope.go:122] "RemoveContainer" containerID="c7b491a1f94b7540bb8bd0aba6f4e2823ae4521ea8a70fd0f3d86374d9c6330b"
Nov 03 20:39:11.056934 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:11.056905    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"]
Nov 03 20:39:11.061167 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:11.061139    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-46a6d7-openshift-default-6fc4757985-24x67"]
Nov 03 20:39:11.376105 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:11.375866    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a884ccd6-a398-44d2-a3a7-40c8e96abd66" path="/var/lib/kubelet/pods/a884ccd6-a398-44d2-a3a7-40c8e96abd66/volumes"
Nov 03 20:39:12.285710 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:12.285669    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd"]
Nov 03 20:39:12.882253 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:12.877480    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6"]
Nov 03 20:39:12.882253 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:12.877729    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6" podUID="92c8f021-930c-438f-baf2-7457458d03af" containerName="maas-api" containerID="cri-o://1313bdce7771ec9e4c62672e9bc2960add3a9ee9b78d5f7f6ab75a8ac080322e" gracePeriod=30
Nov 03 20:39:13.048474 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:13.048441    5591 generic.go:356] "Generic (PLEG): container finished" podID="92c8f021-930c-438f-baf2-7457458d03af" containerID="1313bdce7771ec9e4c62672e9bc2960add3a9ee9b78d5f7f6ab75a8ac080322e" exitCode=0
Nov 03 20:39:13.048660 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:13.048513    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6" event={"ID":"92c8f021-930c-438f-baf2-7457458d03af","Type":"ContainerDied","Data":"1313bdce7771ec9e4c62672e9bc2960add3a9ee9b78d5f7f6ab75a8ac080322e"}
Nov 03 20:39:13.241278 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:13.235278    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6"
Nov 03 20:39:13.307109 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:13.307066    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92c8f021-930c-438f-baf2-7457458d03af-metrics-certs\" (UniqueName: \"kubernetes.io/secret/92c8f021-930c-438f-baf2-7457458d03af-metrics-certs\") pod \"92c8f021-930c-438f-baf2-7457458d03af\" (UID: \"92c8f021-930c-438f-baf2-7457458d03af\") "
Nov 03 20:39:13.307680 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:13.307195    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92c8f021-930c-438f-baf2-7457458d03af-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92c8f021-930c-438f-baf2-7457458d03af-maas-api-tls\") pod \"92c8f021-930c-438f-baf2-7457458d03af\" (UID: \"92c8f021-930c-438f-baf2-7457458d03af\") "
Nov 03 20:39:13.307680 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:13.307333    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92c8f021-930c-438f-baf2-7457458d03af-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c8f021-930c-438f-baf2-7457458d03af-openshift-service-ca\") pod \"92c8f021-930c-438f-baf2-7457458d03af\" (UID: \"92c8f021-930c-438f-baf2-7457458d03af\") "
Nov 03 20:39:13.307680 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:13.307453    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92c8f021-930c-438f-baf2-7457458d03af-kube-api-access-tqhrf\" (UniqueName: \"kubernetes.io/projected/92c8f021-930c-438f-baf2-7457458d03af-kube-api-access-tqhrf\") pod \"92c8f021-930c-438f-baf2-7457458d03af\" (UID: \"92c8f021-930c-438f-baf2-7457458d03af\") "
Nov 03 20:39:13.307893 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:13.307727    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92c8f021-930c-438f-baf2-7457458d03af-openshift-service-ca" pod "92c8f021-930c-438f-baf2-7457458d03af" (UID: "92c8f021-930c-438f-baf2-7457458d03af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:13.308454 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:13.308425    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c8f021-930c-438f-baf2-7457458d03af-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:13.310081 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:13.310057    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92c8f021-930c-438f-baf2-7457458d03af-maas-api-tls" pod "92c8f021-930c-438f-baf2-7457458d03af" (UID: "92c8f021-930c-438f-baf2-7457458d03af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:13.310613 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:13.310584    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c8f021-930c-438f-baf2-7457458d03af-kube-api-access-tqhrf" pod "92c8f021-930c-438f-baf2-7457458d03af" (UID: "92c8f021-930c-438f-baf2-7457458d03af"). InnerVolumeSpecName "kube-api-access-tqhrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:13.311971 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:13.311940    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92c8f021-930c-438f-baf2-7457458d03af-metrics-certs" pod "92c8f021-930c-438f-baf2-7457458d03af" (UID: "92c8f021-930c-438f-baf2-7457458d03af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:13.409701 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:13.409615    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92c8f021-930c-438f-baf2-7457458d03af-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:13.409701 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:13.409646    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92c8f021-930c-438f-baf2-7457458d03af-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:13.409701 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:13.409657    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqhrf\" (UniqueName: \"kubernetes.io/projected/92c8f021-930c-438f-baf2-7457458d03af-kube-api-access-tqhrf\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:14.056874 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:14.056841    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6"
Nov 03 20:39:14.057055 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:14.056869    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6" event={"ID":"92c8f021-930c-438f-baf2-7457458d03af","Type":"ContainerDied","Data":"fb58a7271b2bcd087dd5a8f3f75a7ae5da351c2f5c67d1499f764301b59f24c0"}
Nov 03 20:39:14.057055 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:14.056938    5591 scope.go:122] "RemoveContainer" containerID="1313bdce7771ec9e4c62672e9bc2960add3a9ee9b78d5f7f6ab75a8ac080322e"
Nov 03 20:39:14.076013 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:14.075985    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6"]
Nov 03 20:39:14.079882 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:14.079851    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-33e267-8644cc9bc5-vrbf6"]
Nov 03 20:39:15.352149 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:15.352118    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92c8f021-930c-438f-baf2-7457458d03af" path="/var/lib/kubelet/pods/92c8f021-930c-438f-baf2-7457458d03af/volumes"
Nov 03 20:39:16.600099 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:16.600061    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27"]
Nov 03 20:39:16.615672 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:16.615638    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm"]
Nov 03 20:39:17.287503 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:17.287433    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd" podUID="d26da545-16eb-415b-91ed-63db42e8f01d" containerName="payload-pre-processing" containerID="cri-o://fa9e54a5b0138583280047135d5e5a3af0c2e887f2778e10ecf1a4800b290474" gracePeriod=25
Nov 03 20:39:18.272229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:18.267357    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k"]
Nov 03 20:39:18.272229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:18.267614    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k" podUID="d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb" containerName="maas-api" containerID="cri-o://401474caa9d654980d599143a885653f2c28b2a3b48d2025ad6fa707e252184f" gracePeriod=30
Nov 03 20:39:18.692541 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:18.692513    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k"
Nov 03 20:39:18.808079 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:18.808049    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-openshift-service-ca\") pod \"d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb\" (UID: \"d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb\") "
Nov 03 20:39:18.808272 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:18.808104    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-maas-api-tls\") pod \"d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb\" (UID: \"d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb\") "
Nov 03 20:39:18.808272 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:18.808158    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-kube-api-access-tqhxq\" (UniqueName: \"kubernetes.io/projected/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-kube-api-access-tqhxq\") pod \"d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb\" (UID: \"d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb\") "
Nov 03 20:39:18.808420 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:18.808369    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-metrics-certs\") pod \"d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb\" (UID: \"d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb\") "
Nov 03 20:39:18.808537 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:18.808475    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-openshift-service-ca" pod "d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb" (UID: "d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:18.808903 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:18.808884    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:18.810587 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:18.810557    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-metrics-certs" pod "d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb" (UID: "d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:18.811000 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:18.810980    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-kube-api-access-tqhxq" pod "d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb" (UID: "d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb"). InnerVolumeSpecName "kube-api-access-tqhxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:18.811085 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:18.811001    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-maas-api-tls" pod "d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb" (UID: "d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:18.910848 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:18.910766    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:18.910848 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:18.910800    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:18.910848 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:18.910811    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqhxq\" (UniqueName: \"kubernetes.io/projected/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb-kube-api-access-tqhxq\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:19.095563 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:19.095528    5591 generic.go:356] "Generic (PLEG): container finished" podID="d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb" containerID="401474caa9d654980d599143a885653f2c28b2a3b48d2025ad6fa707e252184f" exitCode=0
Nov 03 20:39:19.095780 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:19.095587    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k"
Nov 03 20:39:19.095780 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:19.095611    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k" event={"ID":"d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb","Type":"ContainerDied","Data":"401474caa9d654980d599143a885653f2c28b2a3b48d2025ad6fa707e252184f"}
Nov 03 20:39:19.095780 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:19.095660    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k" event={"ID":"d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb","Type":"ContainerDied","Data":"3062b68738b2fe5d1e03d5c7898b15beb1bd117eefdbd4ed0f87a2a4504d9d51"}
Nov 03 20:39:19.095780 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:19.095682    5591 scope.go:122] "RemoveContainer" containerID="401474caa9d654980d599143a885653f2c28b2a3b48d2025ad6fa707e252184f"
Nov 03 20:39:19.120175 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:19.120155    5591 scope.go:122] "RemoveContainer" containerID="401474caa9d654980d599143a885653f2c28b2a3b48d2025ad6fa707e252184f"
Nov 03 20:39:19.120499 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:39:19.120477    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"401474caa9d654980d599143a885653f2c28b2a3b48d2025ad6fa707e252184f\": container with ID starting with 401474caa9d654980d599143a885653f2c28b2a3b48d2025ad6fa707e252184f not found: ID does not exist" containerID="401474caa9d654980d599143a885653f2c28b2a3b48d2025ad6fa707e252184f"
Nov 03 20:39:19.120548 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:19.120508    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"401474caa9d654980d599143a885653f2c28b2a3b48d2025ad6fa707e252184f"} err="failed to get container status \"401474caa9d654980d599143a885653f2c28b2a3b48d2025ad6fa707e252184f\": rpc error: code = NotFound desc = could not find container \"401474caa9d654980d599143a885653f2c28b2a3b48d2025ad6fa707e252184f\": container with ID starting with 401474caa9d654980d599143a885653f2c28b2a3b48d2025ad6fa707e252184f not found: ID does not exist"
Nov 03 20:39:19.130829 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:19.130797    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k"]
Nov 03 20:39:19.135565 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:19.135540    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-836265-74c877b94f-v2k4k"]
Nov 03 20:39:19.350804 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:19.350765    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb" path="/var/lib/kubelet/pods/d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb/volumes"
Nov 03 20:39:21.610941 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:21.610898    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27" podUID="a67dbe17-4521-40e5-b4aa-602094676670" containerName="payload-processing" containerID="cri-o://58ef0ded3606a7fd1d51409e3e3fb3e23260372f4fb4fdee41f07e7888de98d1" gracePeriod=25
Nov 03 20:39:21.616348 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:21.616307    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm" podUID="fe2c855d-4f89-4546-b580-e5f7e22eb9bc" containerName="payload-pre-processing" containerID="cri-o://9535fd95d2bd30b629c9e2f4735293bbbb279ec784730475cbbe7b987f192942" gracePeriod=25
Nov 03 20:39:21.982336 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:21.982303    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27"
Nov 03 20:39:22.020889 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:22.020848    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf"]
Nov 03 20:39:22.086803 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:22.086764    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a67dbe17-4521-40e5-b4aa-602094676670-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67dbe17-4521-40e5-b4aa-602094676670-plugins-config-volume\") pod \"a67dbe17-4521-40e5-b4aa-602094676670\" (UID: \"a67dbe17-4521-40e5-b4aa-602094676670\") "
Nov 03 20:39:22.087022 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:22.086944    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67dbe17-4521-40e5-b4aa-602094676670-kube-api-access-snvgp\" (UniqueName: \"kubernetes.io/projected/a67dbe17-4521-40e5-b4aa-602094676670-kube-api-access-snvgp\") pod \"a67dbe17-4521-40e5-b4aa-602094676670\" (UID: \"a67dbe17-4521-40e5-b4aa-602094676670\") "
Nov 03 20:39:22.091228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:22.087361    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a67dbe17-4521-40e5-b4aa-602094676670-plugins-config-volume" pod "a67dbe17-4521-40e5-b4aa-602094676670" (UID: "a67dbe17-4521-40e5-b4aa-602094676670"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:22.091228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:22.087729    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a67dbe17-4521-40e5-b4aa-602094676670-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:22.100080 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:22.100045    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67dbe17-4521-40e5-b4aa-602094676670-kube-api-access-snvgp" pod "a67dbe17-4521-40e5-b4aa-602094676670" (UID: "a67dbe17-4521-40e5-b4aa-602094676670"). InnerVolumeSpecName "kube-api-access-snvgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:22.128427 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:22.125390    5591 generic.go:356] "Generic (PLEG): container finished" podID="a67dbe17-4521-40e5-b4aa-602094676670" containerID="58ef0ded3606a7fd1d51409e3e3fb3e23260372f4fb4fdee41f07e7888de98d1" exitCode=0
Nov 03 20:39:22.128427 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:22.125759    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27" event={"ID":"a67dbe17-4521-40e5-b4aa-602094676670","Type":"ContainerDied","Data":"58ef0ded3606a7fd1d51409e3e3fb3e23260372f4fb4fdee41f07e7888de98d1"}
Nov 03 20:39:22.128427 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:22.125806    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27" event={"ID":"a67dbe17-4521-40e5-b4aa-602094676670","Type":"ContainerDied","Data":"6489c477db794c5c1819716e21375e2dc49786251f28b44d39155edccf3870d9"}
Nov 03 20:39:22.128427 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:22.125827    5591 scope.go:122] "RemoveContainer" containerID="58ef0ded3606a7fd1d51409e3e3fb3e23260372f4fb4fdee41f07e7888de98d1"
Nov 03 20:39:22.128427 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:22.125916    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27"
Nov 03 20:39:22.144089 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:22.144070    5591 scope.go:122] "RemoveContainer" containerID="58ef0ded3606a7fd1d51409e3e3fb3e23260372f4fb4fdee41f07e7888de98d1"
Nov 03 20:39:22.145261 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:39:22.145223    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58ef0ded3606a7fd1d51409e3e3fb3e23260372f4fb4fdee41f07e7888de98d1\": container with ID starting with 58ef0ded3606a7fd1d51409e3e3fb3e23260372f4fb4fdee41f07e7888de98d1 not found: ID does not exist" containerID="58ef0ded3606a7fd1d51409e3e3fb3e23260372f4fb4fdee41f07e7888de98d1"
Nov 03 20:39:22.145384 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:22.145269    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58ef0ded3606a7fd1d51409e3e3fb3e23260372f4fb4fdee41f07e7888de98d1"} err="failed to get container status \"58ef0ded3606a7fd1d51409e3e3fb3e23260372f4fb4fdee41f07e7888de98d1\": rpc error: code = NotFound desc = could not find container \"58ef0ded3606a7fd1d51409e3e3fb3e23260372f4fb4fdee41f07e7888de98d1\": container with ID starting with 58ef0ded3606a7fd1d51409e3e3fb3e23260372f4fb4fdee41f07e7888de98d1 not found: ID does not exist"
Nov 03 20:39:22.153381 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:22.153358    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27"]
Nov 03 20:39:22.159040 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:22.159016    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8b7351-5fc98886d-twv27"]
Nov 03 20:39:22.190193 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:22.190165    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snvgp\" (UniqueName: \"kubernetes.io/projected/a67dbe17-4521-40e5-b4aa-602094676670-kube-api-access-snvgp\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:23.350593 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:23.350559    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a67dbe17-4521-40e5-b4aa-602094676670" path="/var/lib/kubelet/pods/a67dbe17-4521-40e5-b4aa-602094676670/volumes"
Nov 03 20:39:23.511279 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:23.507894    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4"]
Nov 03 20:39:23.511279 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:23.508127    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4" podUID="fbb496cd-b12d-4331-af8b-417c6fbffdf2" containerName="maas-api" containerID="cri-o://c236dd44ac0c3e5f98212c5d973bb16a0285ca834b2df1294d7e513583fad17a" gracePeriod=30
Nov 03 20:39:23.907045 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:23.907013    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4"
Nov 03 20:39:23.939882 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:23.939846    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbb496cd-b12d-4331-af8b-417c6fbffdf2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb496cd-b12d-4331-af8b-417c6fbffdf2-openshift-service-ca\") pod \"fbb496cd-b12d-4331-af8b-417c6fbffdf2\" (UID: \"fbb496cd-b12d-4331-af8b-417c6fbffdf2\") "
Nov 03 20:39:23.940101 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:23.939974    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbb496cd-b12d-4331-af8b-417c6fbffdf2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbb496cd-b12d-4331-af8b-417c6fbffdf2-metrics-certs\") pod \"fbb496cd-b12d-4331-af8b-417c6fbffdf2\" (UID: \"fbb496cd-b12d-4331-af8b-417c6fbffdf2\") "
Nov 03 20:39:23.940101 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:23.940096    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbb496cd-b12d-4331-af8b-417c6fbffdf2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb496cd-b12d-4331-af8b-417c6fbffdf2-maas-api-tls\") pod \"fbb496cd-b12d-4331-af8b-417c6fbffdf2\" (UID: \"fbb496cd-b12d-4331-af8b-417c6fbffdf2\") "
Nov 03 20:39:23.940328 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:23.940308    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb496cd-b12d-4331-af8b-417c6fbffdf2-kube-api-access-szxsb\" (UniqueName: \"kubernetes.io/projected/fbb496cd-b12d-4331-af8b-417c6fbffdf2-kube-api-access-szxsb\") pod \"fbb496cd-b12d-4331-af8b-417c6fbffdf2\" (UID: \"fbb496cd-b12d-4331-af8b-417c6fbffdf2\") "
Nov 03 20:39:23.946793 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:23.944024    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb496cd-b12d-4331-af8b-417c6fbffdf2-openshift-service-ca" pod "fbb496cd-b12d-4331-af8b-417c6fbffdf2" (UID: "fbb496cd-b12d-4331-af8b-417c6fbffdf2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:23.976237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:23.969557    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbb496cd-b12d-4331-af8b-417c6fbffdf2-metrics-certs" pod "fbb496cd-b12d-4331-af8b-417c6fbffdf2" (UID: "fbb496cd-b12d-4331-af8b-417c6fbffdf2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:23.976237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:23.969578    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbb496cd-b12d-4331-af8b-417c6fbffdf2-maas-api-tls" pod "fbb496cd-b12d-4331-af8b-417c6fbffdf2" (UID: "fbb496cd-b12d-4331-af8b-417c6fbffdf2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:23.976237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:23.975981    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb496cd-b12d-4331-af8b-417c6fbffdf2-kube-api-access-szxsb" pod "fbb496cd-b12d-4331-af8b-417c6fbffdf2" (UID: "fbb496cd-b12d-4331-af8b-417c6fbffdf2"). InnerVolumeSpecName "kube-api-access-szxsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:24.043677 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:24.043640    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbb496cd-b12d-4331-af8b-417c6fbffdf2-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:24.043677 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:24.043682    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szxsb\" (UniqueName: \"kubernetes.io/projected/fbb496cd-b12d-4331-af8b-417c6fbffdf2-kube-api-access-szxsb\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:24.043914 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:24.043699    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbb496cd-b12d-4331-af8b-417c6fbffdf2-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:24.043914 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:24.043714    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbb496cd-b12d-4331-af8b-417c6fbffdf2-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:24.150072 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:24.149965    5591 generic.go:356] "Generic (PLEG): container finished" podID="fbb496cd-b12d-4331-af8b-417c6fbffdf2" containerID="c236dd44ac0c3e5f98212c5d973bb16a0285ca834b2df1294d7e513583fad17a" exitCode=0
Nov 03 20:39:24.150072 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:24.150031    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4" event={"ID":"fbb496cd-b12d-4331-af8b-417c6fbffdf2","Type":"ContainerDied","Data":"c236dd44ac0c3e5f98212c5d973bb16a0285ca834b2df1294d7e513583fad17a"}
Nov 03 20:39:24.150375 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:24.150127    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4"
Nov 03 20:39:24.150375 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:24.150154    5591 scope.go:122] "RemoveContainer" containerID="c236dd44ac0c3e5f98212c5d973bb16a0285ca834b2df1294d7e513583fad17a"
Nov 03 20:39:24.150375 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:24.150132    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4" event={"ID":"fbb496cd-b12d-4331-af8b-417c6fbffdf2","Type":"ContainerDied","Data":"773f26cc0bb4657b2ecde6a5339b7004827cbbbbadc69d84641cca2c0b0fa1f1"}
Nov 03 20:39:24.162080 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:24.162060    5591 scope.go:122] "RemoveContainer" containerID="c236dd44ac0c3e5f98212c5d973bb16a0285ca834b2df1294d7e513583fad17a"
Nov 03 20:39:24.162378 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:39:24.162359    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c236dd44ac0c3e5f98212c5d973bb16a0285ca834b2df1294d7e513583fad17a\": container with ID starting with c236dd44ac0c3e5f98212c5d973bb16a0285ca834b2df1294d7e513583fad17a not found: ID does not exist" containerID="c236dd44ac0c3e5f98212c5d973bb16a0285ca834b2df1294d7e513583fad17a"
Nov 03 20:39:24.162435 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:24.162388    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c236dd44ac0c3e5f98212c5d973bb16a0285ca834b2df1294d7e513583fad17a"} err="failed to get container status \"c236dd44ac0c3e5f98212c5d973bb16a0285ca834b2df1294d7e513583fad17a\": rpc error: code = NotFound desc = could not find container \"c236dd44ac0c3e5f98212c5d973bb16a0285ca834b2df1294d7e513583fad17a\": container with ID starting with c236dd44ac0c3e5f98212c5d973bb16a0285ca834b2df1294d7e513583fad17a not found: ID does not exist"
Nov 03 20:39:24.177647 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:24.177618    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4"]
Nov 03 20:39:24.181996 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:24.181971    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8b7351-89cbdb479-f5jj4"]
Nov 03 20:39:25.215512 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:25.215467    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd" podUID="d26da545-16eb-415b-91ed-63db42e8f01d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:39:25.350738 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:25.350707    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb496cd-b12d-4331-af8b-417c6fbffdf2" path="/var/lib/kubelet/pods/fbb496cd-b12d-4331-af8b-417c6fbffdf2/volumes"
Nov 03 20:39:25.830626 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:25.830570    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln"]
Nov 03 20:39:25.831461 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:25.831195    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln" podUID="86168259-d196-495e-9435-33d4f0e625d9" containerName="maas-api" containerID="cri-o://0d95a0f4c09a0e24cdfc1de8d3a706147ebab212c830ad14ac999d77db8fb133" gracePeriod=30
Nov 03 20:39:25.944384 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:25.944348    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm" podUID="fe2c855d-4f89-4546-b580-e5f7e22eb9bc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:39:26.203595 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:26.203565    5591 generic.go:356] "Generic (PLEG): container finished" podID="86168259-d196-495e-9435-33d4f0e625d9" containerID="0d95a0f4c09a0e24cdfc1de8d3a706147ebab212c830ad14ac999d77db8fb133" exitCode=0
Nov 03 20:39:26.203782 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:26.203636    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln" event={"ID":"86168259-d196-495e-9435-33d4f0e625d9","Type":"ContainerDied","Data":"0d95a0f4c09a0e24cdfc1de8d3a706147ebab212c830ad14ac999d77db8fb133"}
Nov 03 20:39:26.238092 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:26.238065    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln"
Nov 03 20:39:26.281644 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:26.281616    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86168259-d196-495e-9435-33d4f0e625d9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86168259-d196-495e-9435-33d4f0e625d9-openshift-service-ca\") pod \"86168259-d196-495e-9435-33d4f0e625d9\" (UID: \"86168259-d196-495e-9435-33d4f0e625d9\") "
Nov 03 20:39:26.281845 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:26.281676    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86168259-d196-495e-9435-33d4f0e625d9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86168259-d196-495e-9435-33d4f0e625d9-maas-api-tls\") pod \"86168259-d196-495e-9435-33d4f0e625d9\" (UID: \"86168259-d196-495e-9435-33d4f0e625d9\") "
Nov 03 20:39:26.281845 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:26.281733    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86168259-d196-495e-9435-33d4f0e625d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/86168259-d196-495e-9435-33d4f0e625d9-metrics-certs\") pod \"86168259-d196-495e-9435-33d4f0e625d9\" (UID: \"86168259-d196-495e-9435-33d4f0e625d9\") "
Nov 03 20:39:26.281845 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:26.281769    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86168259-d196-495e-9435-33d4f0e625d9-kube-api-access-ztkxt\" (UniqueName: \"kubernetes.io/projected/86168259-d196-495e-9435-33d4f0e625d9-kube-api-access-ztkxt\") pod \"86168259-d196-495e-9435-33d4f0e625d9\" (UID: \"86168259-d196-495e-9435-33d4f0e625d9\") "
Nov 03 20:39:26.282044 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:26.281988    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86168259-d196-495e-9435-33d4f0e625d9-openshift-service-ca" pod "86168259-d196-495e-9435-33d4f0e625d9" (UID: "86168259-d196-495e-9435-33d4f0e625d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:26.282550 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:26.282513    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86168259-d196-495e-9435-33d4f0e625d9-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:26.284186 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:26.284155    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86168259-d196-495e-9435-33d4f0e625d9-kube-api-access-ztkxt" pod "86168259-d196-495e-9435-33d4f0e625d9" (UID: "86168259-d196-495e-9435-33d4f0e625d9"). InnerVolumeSpecName "kube-api-access-ztkxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:26.284552 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:26.284527    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86168259-d196-495e-9435-33d4f0e625d9-maas-api-tls" pod "86168259-d196-495e-9435-33d4f0e625d9" (UID: "86168259-d196-495e-9435-33d4f0e625d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:26.284668 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:26.284612    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86168259-d196-495e-9435-33d4f0e625d9-metrics-certs" pod "86168259-d196-495e-9435-33d4f0e625d9" (UID: "86168259-d196-495e-9435-33d4f0e625d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:26.384024 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:26.383981    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86168259-d196-495e-9435-33d4f0e625d9-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:26.384024 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:26.384017    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86168259-d196-495e-9435-33d4f0e625d9-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:26.384024 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:26.384031    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztkxt\" (UniqueName: \"kubernetes.io/projected/86168259-d196-495e-9435-33d4f0e625d9-kube-api-access-ztkxt\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:27.022136 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:27.022085    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf" podUID="080839df-c38a-4bfc-8b72-008a2c12de63" containerName="payload-processing" containerID="cri-o://bcebf22a6126cceaadded0be8c2e3f0417bb6bb1aba21d966d644853a34ea0cf" gracePeriod=25
Nov 03 20:39:27.222021 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:27.221028    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln"
Nov 03 20:39:27.225803 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:27.225617    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln" event={"ID":"86168259-d196-495e-9435-33d4f0e625d9","Type":"ContainerDied","Data":"6da58e61f9a4a1d5811ae5ee663ca359b4624f52ad916ce31cf17dd4647fb420"}
Nov 03 20:39:27.225803 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:27.225729    5591 scope.go:122] "RemoveContainer" containerID="0d95a0f4c09a0e24cdfc1de8d3a706147ebab212c830ad14ac999d77db8fb133"
Nov 03 20:39:27.228278 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:27.228197    5591 generic.go:356] "Generic (PLEG): container finished" podID="080839df-c38a-4bfc-8b72-008a2c12de63" containerID="bcebf22a6126cceaadded0be8c2e3f0417bb6bb1aba21d966d644853a34ea0cf" exitCode=0
Nov 03 20:39:27.228573 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:27.228511    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf" event={"ID":"080839df-c38a-4bfc-8b72-008a2c12de63","Type":"ContainerDied","Data":"bcebf22a6126cceaadded0be8c2e3f0417bb6bb1aba21d966d644853a34ea0cf"}
Nov 03 20:39:27.253324 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:27.253297    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln"]
Nov 03 20:39:27.258947 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:27.258923    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4c2a28-fcbc7694d-4dsln"]
Nov 03 20:39:27.305870 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:27.305839    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf"
Nov 03 20:39:27.352076 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:27.352033    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86168259-d196-495e-9435-33d4f0e625d9" path="/var/lib/kubelet/pods/86168259-d196-495e-9435-33d4f0e625d9/volumes"
Nov 03 20:39:27.401503 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:27.401473    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/080839df-c38a-4bfc-8b72-008a2c12de63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/080839df-c38a-4bfc-8b72-008a2c12de63-plugins-config-volume\") pod \"080839df-c38a-4bfc-8b72-008a2c12de63\" (UID: \"080839df-c38a-4bfc-8b72-008a2c12de63\") "
Nov 03 20:39:27.401723 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:27.401708    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/080839df-c38a-4bfc-8b72-008a2c12de63-kube-api-access-wbkqb\" (UniqueName: \"kubernetes.io/projected/080839df-c38a-4bfc-8b72-008a2c12de63-kube-api-access-wbkqb\") pod \"080839df-c38a-4bfc-8b72-008a2c12de63\" (UID: \"080839df-c38a-4bfc-8b72-008a2c12de63\") "
Nov 03 20:39:27.401832 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:27.401811    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/080839df-c38a-4bfc-8b72-008a2c12de63-plugins-config-volume" pod "080839df-c38a-4bfc-8b72-008a2c12de63" (UID: "080839df-c38a-4bfc-8b72-008a2c12de63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:27.402245 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:27.402225    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/080839df-c38a-4bfc-8b72-008a2c12de63-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:27.404151 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:27.404127    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080839df-c38a-4bfc-8b72-008a2c12de63-kube-api-access-wbkqb" pod "080839df-c38a-4bfc-8b72-008a2c12de63" (UID: "080839df-c38a-4bfc-8b72-008a2c12de63"). InnerVolumeSpecName "kube-api-access-wbkqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:27.503896 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:27.503851    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbkqb\" (UniqueName: \"kubernetes.io/projected/080839df-c38a-4bfc-8b72-008a2c12de63-kube-api-access-wbkqb\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:28.236980 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:28.236940    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf"
Nov 03 20:39:28.236980 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:28.236965    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf" event={"ID":"080839df-c38a-4bfc-8b72-008a2c12de63","Type":"ContainerDied","Data":"85b279f1e823aedc37df6a27b43663ea81654375d18658960b997e2c492af2dd"}
Nov 03 20:39:28.237226 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:28.237058    5591 scope.go:122] "RemoveContainer" containerID="bcebf22a6126cceaadded0be8c2e3f0417bb6bb1aba21d966d644853a34ea0cf"
Nov 03 20:39:28.262443 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:28.262412    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf"]
Nov 03 20:39:28.267577 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:28.267552    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4c2a28-86b687fd6-z8rqf"]
Nov 03 20:39:28.750319 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:39:28.750093    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:39:29.350048 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:29.350008    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="080839df-c38a-4bfc-8b72-008a2c12de63" path="/var/lib/kubelet/pods/080839df-c38a-4bfc-8b72-008a2c12de63/volumes"
Nov 03 20:39:32.514430 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:32.514391    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"]
Nov 03 20:39:32.514931 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:32.514672    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds" podUID="e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15" containerName="istio-proxy" containerID="cri-o://e4336af13fb5abc05d4709ed8396b95ad3168b3d0662e4d414fcfdfe5ee00913" gracePeriod=30
Nov 03 20:39:32.808555 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:32.808436    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6"]
Nov 03 20:39:32.809128 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:32.809092    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6" podUID="df5c29d5-b63f-47ca-892a-d288dfa08f7f" containerName="maas-api" containerID="cri-o://37f1cfebbd3e07efd3ee56ec46169f142a939b80f299cd3d92ae9fed3cd0873b" gracePeriod=30
Nov 03 20:39:33.190146 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:33.190119    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6"
Nov 03 20:39:33.230404 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:33.230371    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df5c29d5-b63f-47ca-892a-d288dfa08f7f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df5c29d5-b63f-47ca-892a-d288dfa08f7f-openshift-service-ca\") pod \"df5c29d5-b63f-47ca-892a-d288dfa08f7f\" (UID: \"df5c29d5-b63f-47ca-892a-d288dfa08f7f\") "
Nov 03 20:39:33.230613 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:33.230508    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df5c29d5-b63f-47ca-892a-d288dfa08f7f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df5c29d5-b63f-47ca-892a-d288dfa08f7f-maas-api-tls\") pod \"df5c29d5-b63f-47ca-892a-d288dfa08f7f\" (UID: \"df5c29d5-b63f-47ca-892a-d288dfa08f7f\") "
Nov 03 20:39:33.230732 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:33.230709    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df5c29d5-b63f-47ca-892a-d288dfa08f7f-kube-api-access-vthcp\" (UniqueName: \"kubernetes.io/projected/df5c29d5-b63f-47ca-892a-d288dfa08f7f-kube-api-access-vthcp\") pod \"df5c29d5-b63f-47ca-892a-d288dfa08f7f\" (UID: \"df5c29d5-b63f-47ca-892a-d288dfa08f7f\") "
Nov 03 20:39:33.230867 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:33.230770    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df5c29d5-b63f-47ca-892a-d288dfa08f7f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/df5c29d5-b63f-47ca-892a-d288dfa08f7f-metrics-certs\") pod \"df5c29d5-b63f-47ca-892a-d288dfa08f7f\" (UID: \"df5c29d5-b63f-47ca-892a-d288dfa08f7f\") "
Nov 03 20:39:33.230867 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:33.230807    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df5c29d5-b63f-47ca-892a-d288dfa08f7f-openshift-service-ca" pod "df5c29d5-b63f-47ca-892a-d288dfa08f7f" (UID: "df5c29d5-b63f-47ca-892a-d288dfa08f7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:33.231724 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:33.231603    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df5c29d5-b63f-47ca-892a-d288dfa08f7f-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:33.234292 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:33.234262    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df5c29d5-b63f-47ca-892a-d288dfa08f7f-kube-api-access-vthcp" pod "df5c29d5-b63f-47ca-892a-d288dfa08f7f" (UID: "df5c29d5-b63f-47ca-892a-d288dfa08f7f"). InnerVolumeSpecName "kube-api-access-vthcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:33.234952 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:33.234922    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df5c29d5-b63f-47ca-892a-d288dfa08f7f-metrics-certs" pod "df5c29d5-b63f-47ca-892a-d288dfa08f7f" (UID: "df5c29d5-b63f-47ca-892a-d288dfa08f7f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:33.235133 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:33.235107    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df5c29d5-b63f-47ca-892a-d288dfa08f7f-maas-api-tls" pod "df5c29d5-b63f-47ca-892a-d288dfa08f7f" (UID: "df5c29d5-b63f-47ca-892a-d288dfa08f7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:33.295675 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:33.295642    5591 generic.go:356] "Generic (PLEG): container finished" podID="df5c29d5-b63f-47ca-892a-d288dfa08f7f" containerID="37f1cfebbd3e07efd3ee56ec46169f142a939b80f299cd3d92ae9fed3cd0873b" exitCode=0
Nov 03 20:39:33.295841 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:33.295693    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6"
Nov 03 20:39:33.295841 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:33.295729    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6" event={"ID":"df5c29d5-b63f-47ca-892a-d288dfa08f7f","Type":"ContainerDied","Data":"37f1cfebbd3e07efd3ee56ec46169f142a939b80f299cd3d92ae9fed3cd0873b"}
Nov 03 20:39:33.295841 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:33.295781    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6" event={"ID":"df5c29d5-b63f-47ca-892a-d288dfa08f7f","Type":"ContainerDied","Data":"cf688c99b4cfc0cebfe60d72403627d088345e1b9814994a77f5fb153086f986"}
Nov 03 20:39:33.295841 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:33.295806    5591 scope.go:122] "RemoveContainer" containerID="37f1cfebbd3e07efd3ee56ec46169f142a939b80f299cd3d92ae9fed3cd0873b"
Nov 03 20:39:33.310657 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:33.307978    5591 scope.go:122] "RemoveContainer" containerID="37f1cfebbd3e07efd3ee56ec46169f142a939b80f299cd3d92ae9fed3cd0873b"
Nov 03 20:39:33.310657 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:39:33.309668    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37f1cfebbd3e07efd3ee56ec46169f142a939b80f299cd3d92ae9fed3cd0873b\": container with ID starting with 37f1cfebbd3e07efd3ee56ec46169f142a939b80f299cd3d92ae9fed3cd0873b not found: ID does not exist" containerID="37f1cfebbd3e07efd3ee56ec46169f142a939b80f299cd3d92ae9fed3cd0873b"
Nov 03 20:39:33.310657 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:33.309703    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37f1cfebbd3e07efd3ee56ec46169f142a939b80f299cd3d92ae9fed3cd0873b"} err="failed to get container status \"37f1cfebbd3e07efd3ee56ec46169f142a939b80f299cd3d92ae9fed3cd0873b\": rpc error: code = NotFound desc = could not find container \"37f1cfebbd3e07efd3ee56ec46169f142a939b80f299cd3d92ae9fed3cd0873b\": container with ID starting with 37f1cfebbd3e07efd3ee56ec46169f142a939b80f299cd3d92ae9fed3cd0873b not found: ID does not exist"
Nov 03 20:39:33.346283 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:33.329289    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6"]
Nov 03 20:39:33.346283 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:33.332826    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-660d95-84ddc566bb-x7dn6"]
Nov 03 20:39:33.346283 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:33.333808    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df5c29d5-b63f-47ca-892a-d288dfa08f7f-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:33.346283 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:33.333836    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vthcp\" (UniqueName: \"kubernetes.io/projected/df5c29d5-b63f-47ca-892a-d288dfa08f7f-kube-api-access-vthcp\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:33.346283 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:33.333851    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df5c29d5-b63f-47ca-892a-d288dfa08f7f-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:33.357355 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:33.357322    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df5c29d5-b63f-47ca-892a-d288dfa08f7f" path="/var/lib/kubelet/pods/df5c29d5-b63f-47ca-892a-d288dfa08f7f/volumes"
Nov 03 20:39:33.621254 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:33.621221    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:39:35.215167 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:35.215127    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd" podUID="d26da545-16eb-415b-91ed-63db42e8f01d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:39:35.944735 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:35.944688    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm" podUID="fe2c855d-4f89-4546-b580-e5f7e22eb9bc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:39:36.889385 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:36.889348    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z"]
Nov 03 20:39:36.903404 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:36.903373    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt"]
Nov 03 20:39:37.665851 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:37.665813    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd"
Nov 03 20:39:37.666379 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:37.666346    5591 scope.go:122] "RemoveContainer" containerID="fa9e54a5b0138583280047135d5e5a3af0c2e887f2778e10ecf1a4800b290474"
Nov 03 20:39:37.835744 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:37.835663    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d26da545-16eb-415b-91ed-63db42e8f01d-kube-api-access-568sw\" (UniqueName: \"kubernetes.io/projected/d26da545-16eb-415b-91ed-63db42e8f01d-kube-api-access-568sw\") pod \"d26da545-16eb-415b-91ed-63db42e8f01d\" (UID: \"d26da545-16eb-415b-91ed-63db42e8f01d\") "
Nov 03 20:39:37.835906 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:37.835756    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d26da545-16eb-415b-91ed-63db42e8f01d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d26da545-16eb-415b-91ed-63db42e8f01d-plugins-config-volume\") pod \"d26da545-16eb-415b-91ed-63db42e8f01d\" (UID: \"d26da545-16eb-415b-91ed-63db42e8f01d\") "
Nov 03 20:39:37.836116 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:37.836084    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d26da545-16eb-415b-91ed-63db42e8f01d-plugins-config-volume" pod "d26da545-16eb-415b-91ed-63db42e8f01d" (UID: "d26da545-16eb-415b-91ed-63db42e8f01d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:37.836342 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:37.836323    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d26da545-16eb-415b-91ed-63db42e8f01d-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:37.837957 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:37.837932    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26da545-16eb-415b-91ed-63db42e8f01d-kube-api-access-568sw" pod "d26da545-16eb-415b-91ed-63db42e8f01d" (UID: "d26da545-16eb-415b-91ed-63db42e8f01d"). InnerVolumeSpecName "kube-api-access-568sw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:37.875587 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:37.875561    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:39:37.938151 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:37.938114    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-568sw\" (UniqueName: \"kubernetes.io/projected/d26da545-16eb-415b-91ed-63db42e8f01d-kube-api-access-568sw\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:38.039087 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.039057    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-data\") pod \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") "
Nov 03 20:39:38.039305 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.039099    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-workload-socket\") pod \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") "
Nov 03 20:39:38.039305 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.039139    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-kube-api-access-mjg29\" (UniqueName: \"kubernetes.io/projected/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-kube-api-access-mjg29\") pod \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") "
Nov 03 20:39:38.039305 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.039168    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-workload-certs\") pod \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") "
Nov 03 20:39:38.039305 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.039265    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-envoy\") pod \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") "
Nov 03 20:39:38.040391 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.039339    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-token\" (UniqueName: \"kubernetes.io/projected/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-token\") pod \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") "
Nov 03 20:39:38.040391 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.039385    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istiod-ca-cert\") pod \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") "
Nov 03 20:39:38.040391 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.039406    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-data" pod "e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15" (UID: "e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:38.040391 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.039417    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-credential-socket\") pod \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") "
Nov 03 20:39:38.040391 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.039463    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-workload-socket" pod "e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15" (UID: "e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:38.040391 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.039572    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-workload-certs" pod "e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15" (UID: "e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:38.040391 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.039576    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-podinfo\") pod \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\" (UID: \"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15\") "
Nov 03 20:39:38.040391 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.039699    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-credential-socket" pod "e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15" (UID: "e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:38.040391 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.039866    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istiod-ca-cert" pod "e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15" (UID: "e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:38.040391 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.040163    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:38.040391 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.040181    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:38.040391 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.040198    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:38.040391 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.040228    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:38.040391 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.040237    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:38.041672 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.041653    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-envoy" pod "e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15" (UID: "e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:38.041741 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.041702    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-podinfo" pod "e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15" (UID: "e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:39:38.042189 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.042169    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-kube-api-access-mjg29" pod "e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15" (UID: "e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15"). InnerVolumeSpecName "kube-api-access-mjg29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:38.042342 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.042225    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-token" pod "e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15" (UID: "e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:38.141865 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.141766    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjg29\" (UniqueName: \"kubernetes.io/projected/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-kube-api-access-mjg29\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:38.141865 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.141804    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:38.141865 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.141818    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:38.141865 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.141830    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:38.337579 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.337534    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd" event={"ID":"d26da545-16eb-415b-91ed-63db42e8f01d","Type":"ContainerDied","Data":"fa9e54a5b0138583280047135d5e5a3af0c2e887f2778e10ecf1a4800b290474"}
Nov 03 20:39:38.337764 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.337608    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd" event={"ID":"d26da545-16eb-415b-91ed-63db42e8f01d","Type":"ContainerDied","Data":"6f8dd2d909a902830d4089d700d3a95222fcea9e9efdcd1961651c8f756c113e"}
Nov 03 20:39:38.339152 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.339125    5591 generic.go:356] "Generic (PLEG): container finished" podID="e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15" containerID="e4336af13fb5abc05d4709ed8396b95ad3168b3d0662e4d414fcfdfe5ee00913" exitCode=0
Nov 03 20:39:38.339286 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.339180    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"
Nov 03 20:39:38.339286 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.339254    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds" event={"ID":"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15","Type":"ContainerDied","Data":"e4336af13fb5abc05d4709ed8396b95ad3168b3d0662e4d414fcfdfe5ee00913"}
Nov 03 20:39:38.339405 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.339311    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds" event={"ID":"e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15","Type":"ContainerDied","Data":"54e14ad51654fd3019d5df5b67b4b9641a584f06c82098af2194a53d2d0dc643"}
Nov 03 20:39:38.339405 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.339337    5591 scope.go:122] "RemoveContainer" containerID="e4336af13fb5abc05d4709ed8396b95ad3168b3d0662e4d414fcfdfe5ee00913"
Nov 03 20:39:38.339513 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.339417    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd"
Nov 03 20:39:38.351925 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.351907    5591 scope.go:122] "RemoveContainer" containerID="e4336af13fb5abc05d4709ed8396b95ad3168b3d0662e4d414fcfdfe5ee00913"
Nov 03 20:39:38.352187 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:39:38.352168    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4336af13fb5abc05d4709ed8396b95ad3168b3d0662e4d414fcfdfe5ee00913\": container with ID starting with e4336af13fb5abc05d4709ed8396b95ad3168b3d0662e4d414fcfdfe5ee00913 not found: ID does not exist" containerID="e4336af13fb5abc05d4709ed8396b95ad3168b3d0662e4d414fcfdfe5ee00913"
Nov 03 20:39:38.352255 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.352197    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4336af13fb5abc05d4709ed8396b95ad3168b3d0662e4d414fcfdfe5ee00913"} err="failed to get container status \"e4336af13fb5abc05d4709ed8396b95ad3168b3d0662e4d414fcfdfe5ee00913\": rpc error: code = NotFound desc = could not find container \"e4336af13fb5abc05d4709ed8396b95ad3168b3d0662e4d414fcfdfe5ee00913\": container with ID starting with e4336af13fb5abc05d4709ed8396b95ad3168b3d0662e4d414fcfdfe5ee00913 not found: ID does not exist"
Nov 03 20:39:38.394445 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.394352    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"]
Nov 03 20:39:38.398739 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.398711    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-8b7351-openshift-default-54db9b4bf7-jsgds"]
Nov 03 20:39:38.409783 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.409747    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd"]
Nov 03 20:39:38.415560 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:38.415529    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-836265-855b654cbc-6rlrd"]
Nov 03 20:39:39.353098 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:39.353067    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d26da545-16eb-415b-91ed-63db42e8f01d" path="/var/lib/kubelet/pods/d26da545-16eb-415b-91ed-63db42e8f01d/volumes"
Nov 03 20:39:39.353727 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:39.353706    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15" path="/var/lib/kubelet/pods/e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15/volumes"
Nov 03 20:39:41.315896 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:41.315858    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h"]
Nov 03 20:39:41.316408 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:41.316155    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h" podUID="d1031c5e-2ece-4213-ba23-33126946df5a" containerName="maas-api" containerID="cri-o://b2b2ea9a9df5580537698cb2fdfb2917a47d849d546f61807ece727563d98317" gracePeriod=30
Nov 03 20:39:41.619806 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:41.619659    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"]
Nov 03 20:39:41.631458 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:41.629118    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"]
Nov 03 20:39:41.755523 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:41.755496    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h"
Nov 03 20:39:41.808652 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:41.805750    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1031c5e-2ece-4213-ba23-33126946df5a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1031c5e-2ece-4213-ba23-33126946df5a-maas-api-tls\") pod \"d1031c5e-2ece-4213-ba23-33126946df5a\" (UID: \"d1031c5e-2ece-4213-ba23-33126946df5a\") "
Nov 03 20:39:41.808652 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:41.805869    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1031c5e-2ece-4213-ba23-33126946df5a-kube-api-access-d7mfx\" (UniqueName: \"kubernetes.io/projected/d1031c5e-2ece-4213-ba23-33126946df5a-kube-api-access-d7mfx\") pod \"d1031c5e-2ece-4213-ba23-33126946df5a\" (UID: \"d1031c5e-2ece-4213-ba23-33126946df5a\") "
Nov 03 20:39:41.808652 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:41.805920    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1031c5e-2ece-4213-ba23-33126946df5a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1031c5e-2ece-4213-ba23-33126946df5a-openshift-service-ca\") pod \"d1031c5e-2ece-4213-ba23-33126946df5a\" (UID: \"d1031c5e-2ece-4213-ba23-33126946df5a\") "
Nov 03 20:39:41.808652 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:41.806068    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1031c5e-2ece-4213-ba23-33126946df5a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1031c5e-2ece-4213-ba23-33126946df5a-metrics-certs\") pod \"d1031c5e-2ece-4213-ba23-33126946df5a\" (UID: \"d1031c5e-2ece-4213-ba23-33126946df5a\") "
Nov 03 20:39:41.808652 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:41.808345    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1031c5e-2ece-4213-ba23-33126946df5a-openshift-service-ca" pod "d1031c5e-2ece-4213-ba23-33126946df5a" (UID: "d1031c5e-2ece-4213-ba23-33126946df5a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:41.836809 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:41.836743    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1031c5e-2ece-4213-ba23-33126946df5a-kube-api-access-d7mfx" pod "d1031c5e-2ece-4213-ba23-33126946df5a" (UID: "d1031c5e-2ece-4213-ba23-33126946df5a"). InnerVolumeSpecName "kube-api-access-d7mfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:41.837544 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:41.837520    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1031c5e-2ece-4213-ba23-33126946df5a-maas-api-tls" pod "d1031c5e-2ece-4213-ba23-33126946df5a" (UID: "d1031c5e-2ece-4213-ba23-33126946df5a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:41.837851 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:41.837819    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1031c5e-2ece-4213-ba23-33126946df5a-metrics-certs" pod "d1031c5e-2ece-4213-ba23-33126946df5a" (UID: "d1031c5e-2ece-4213-ba23-33126946df5a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:41.890035 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:41.889886    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z" podUID="7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455" containerName="payload-processing" containerID="cri-o://e3603e074b6bb8f82aa3597210b84d4c52c34db8d6eaf506ec72159a16bb9ca8" gracePeriod=25
Nov 03 20:39:41.904642 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:41.904595    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt" podUID="c9b4867c-02bd-4c72-8a46-6db1867a40c3" containerName="payload-pre-processing" containerID="cri-o://6d38f24d47ec5959d41f17a76cca7f7e4088ee4c4789f1f21a34842c36536563" gracePeriod=25
Nov 03 20:39:41.910473 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:41.910139    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1031c5e-2ece-4213-ba23-33126946df5a-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:41.910473 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:41.910176    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1031c5e-2ece-4213-ba23-33126946df5a-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:41.910473 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:41.910194    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7mfx\" (UniqueName: \"kubernetes.io/projected/d1031c5e-2ece-4213-ba23-33126946df5a-kube-api-access-d7mfx\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:41.910473 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:41.910227    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1031c5e-2ece-4213-ba23-33126946df5a-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:42.020531 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.020504    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm"
Nov 03 20:39:42.112703 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.112672    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe2c855d-4f89-4546-b580-e5f7e22eb9bc-kube-api-access-6c2m5\" (UniqueName: \"kubernetes.io/projected/fe2c855d-4f89-4546-b580-e5f7e22eb9bc-kube-api-access-6c2m5\") pod \"fe2c855d-4f89-4546-b580-e5f7e22eb9bc\" (UID: \"fe2c855d-4f89-4546-b580-e5f7e22eb9bc\") "
Nov 03 20:39:42.112890 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.112722    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe2c855d-4f89-4546-b580-e5f7e22eb9bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2c855d-4f89-4546-b580-e5f7e22eb9bc-plugins-config-volume\") pod \"fe2c855d-4f89-4546-b580-e5f7e22eb9bc\" (UID: \"fe2c855d-4f89-4546-b580-e5f7e22eb9bc\") "
Nov 03 20:39:42.113804 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.113772    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe2c855d-4f89-4546-b580-e5f7e22eb9bc-plugins-config-volume" pod "fe2c855d-4f89-4546-b580-e5f7e22eb9bc" (UID: "fe2c855d-4f89-4546-b580-e5f7e22eb9bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:42.115997 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.115962    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe2c855d-4f89-4546-b580-e5f7e22eb9bc-kube-api-access-6c2m5" pod "fe2c855d-4f89-4546-b580-e5f7e22eb9bc" (UID: "fe2c855d-4f89-4546-b580-e5f7e22eb9bc"). InnerVolumeSpecName "kube-api-access-6c2m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:42.214260 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.214192    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c2m5\" (UniqueName: \"kubernetes.io/projected/fe2c855d-4f89-4546-b580-e5f7e22eb9bc-kube-api-access-6c2m5\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:42.214260 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.214239    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2c855d-4f89-4546-b580-e5f7e22eb9bc-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:42.217653 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.217618    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z"
Nov 03 20:39:42.249462 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.249429    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt"
Nov 03 20:39:42.315462 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.315427    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455-kube-api-access-fx4j4\" (UniqueName: \"kubernetes.io/projected/7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455-kube-api-access-fx4j4\") pod \"7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455\" (UID: \"7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455\") "
Nov 03 20:39:42.315676 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.315497    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9b4867c-02bd-4c72-8a46-6db1867a40c3-kube-api-access-8p9rh\" (UniqueName: \"kubernetes.io/projected/c9b4867c-02bd-4c72-8a46-6db1867a40c3-kube-api-access-8p9rh\") pod \"c9b4867c-02bd-4c72-8a46-6db1867a40c3\" (UID: \"c9b4867c-02bd-4c72-8a46-6db1867a40c3\") "
Nov 03 20:39:42.315676 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.315536    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455-plugins-config-volume\") pod \"7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455\" (UID: \"7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455\") "
Nov 03 20:39:42.315676 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.315562    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9b4867c-02bd-4c72-8a46-6db1867a40c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b4867c-02bd-4c72-8a46-6db1867a40c3-plugins-config-volume\") pod \"c9b4867c-02bd-4c72-8a46-6db1867a40c3\" (UID: \"c9b4867c-02bd-4c72-8a46-6db1867a40c3\") "
Nov 03 20:39:42.315979 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.315957    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455-plugins-config-volume" pod "7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455" (UID: "7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:42.316326 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.316020    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b4867c-02bd-4c72-8a46-6db1867a40c3-plugins-config-volume" pod "c9b4867c-02bd-4c72-8a46-6db1867a40c3" (UID: "c9b4867c-02bd-4c72-8a46-6db1867a40c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:42.316326 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.316155    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:42.316326 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.316175    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b4867c-02bd-4c72-8a46-6db1867a40c3-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:42.317785 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.317761    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455-kube-api-access-fx4j4" pod "7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455" (UID: "7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455"). InnerVolumeSpecName "kube-api-access-fx4j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:42.317883 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.317866    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b4867c-02bd-4c72-8a46-6db1867a40c3-kube-api-access-8p9rh" pod "c9b4867c-02bd-4c72-8a46-6db1867a40c3" (UID: "c9b4867c-02bd-4c72-8a46-6db1867a40c3"). InnerVolumeSpecName "kube-api-access-8p9rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:42.379920 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.379878    5591 generic.go:356] "Generic (PLEG): container finished" podID="7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455" containerID="e3603e074b6bb8f82aa3597210b84d4c52c34db8d6eaf506ec72159a16bb9ca8" exitCode=0
Nov 03 20:39:42.380113 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.380038    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z" event={"ID":"7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455","Type":"ContainerDied","Data":"e3603e074b6bb8f82aa3597210b84d4c52c34db8d6eaf506ec72159a16bb9ca8"}
Nov 03 20:39:42.380113 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.380088    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z" event={"ID":"7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455","Type":"ContainerDied","Data":"fbed17e06773c6c8b795ccf1fb0d5cac272155c28a1649cd61a2b5e0d4c311c1"}
Nov 03 20:39:42.380293 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.380117    5591 scope.go:122] "RemoveContainer" containerID="e3603e074b6bb8f82aa3597210b84d4c52c34db8d6eaf506ec72159a16bb9ca8"
Nov 03 20:39:42.380293 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.380277    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z"
Nov 03 20:39:42.386626 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.386596    5591 generic.go:356] "Generic (PLEG): container finished" podID="d1031c5e-2ece-4213-ba23-33126946df5a" containerID="b2b2ea9a9df5580537698cb2fdfb2917a47d849d546f61807ece727563d98317" exitCode=0
Nov 03 20:39:42.386752 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.386731    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h" event={"ID":"d1031c5e-2ece-4213-ba23-33126946df5a","Type":"ContainerDied","Data":"b2b2ea9a9df5580537698cb2fdfb2917a47d849d546f61807ece727563d98317"}
Nov 03 20:39:42.386839 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.386789    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h" event={"ID":"d1031c5e-2ece-4213-ba23-33126946df5a","Type":"ContainerDied","Data":"033cfedb9cf8d8889f800fb09b2f37a29fce577128987462e413343d4925914a"}
Nov 03 20:39:42.386914 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.386873    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h"
Nov 03 20:39:42.392257 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.388910    5591 generic.go:356] "Generic (PLEG): container finished" podID="fe2c855d-4f89-4546-b580-e5f7e22eb9bc" containerID="9535fd95d2bd30b629c9e2f4735293bbbb279ec784730475cbbe7b987f192942" exitCode=0
Nov 03 20:39:42.392257 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.389077    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm"
Nov 03 20:39:42.392257 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.390823    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm" event={"ID":"fe2c855d-4f89-4546-b580-e5f7e22eb9bc","Type":"ContainerDied","Data":"9535fd95d2bd30b629c9e2f4735293bbbb279ec784730475cbbe7b987f192942"}
Nov 03 20:39:42.392257 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.390865    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm" event={"ID":"fe2c855d-4f89-4546-b580-e5f7e22eb9bc","Type":"ContainerDied","Data":"c83117b5d799e654d4c3c14fd1ee15149d618c0d71d457540f10cff89b118c62"}
Nov 03 20:39:42.394172 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.394140    5591 generic.go:356] "Generic (PLEG): container finished" podID="c9b4867c-02bd-4c72-8a46-6db1867a40c3" containerID="6d38f24d47ec5959d41f17a76cca7f7e4088ee4c4789f1f21a34842c36536563" exitCode=0
Nov 03 20:39:42.394416 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.394386    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt" event={"ID":"c9b4867c-02bd-4c72-8a46-6db1867a40c3","Type":"ContainerDied","Data":"6d38f24d47ec5959d41f17a76cca7f7e4088ee4c4789f1f21a34842c36536563"}
Nov 03 20:39:42.394504 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.394434    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt" event={"ID":"c9b4867c-02bd-4c72-8a46-6db1867a40c3","Type":"ContainerDied","Data":"e668bb0811bb078076f862c350947c483359042ae6199a237e361ee0aeec76b7"}
Nov 03 20:39:42.394568 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.394548    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt"
Nov 03 20:39:42.397903 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.397883    5591 scope.go:122] "RemoveContainer" containerID="e3603e074b6bb8f82aa3597210b84d4c52c34db8d6eaf506ec72159a16bb9ca8"
Nov 03 20:39:42.398453 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:39:42.398429    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3603e074b6bb8f82aa3597210b84d4c52c34db8d6eaf506ec72159a16bb9ca8\": container with ID starting with e3603e074b6bb8f82aa3597210b84d4c52c34db8d6eaf506ec72159a16bb9ca8 not found: ID does not exist" containerID="e3603e074b6bb8f82aa3597210b84d4c52c34db8d6eaf506ec72159a16bb9ca8"
Nov 03 20:39:42.398632 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.398574    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3603e074b6bb8f82aa3597210b84d4c52c34db8d6eaf506ec72159a16bb9ca8"} err="failed to get container status \"e3603e074b6bb8f82aa3597210b84d4c52c34db8d6eaf506ec72159a16bb9ca8\": rpc error: code = NotFound desc = could not find container \"e3603e074b6bb8f82aa3597210b84d4c52c34db8d6eaf506ec72159a16bb9ca8\": container with ID starting with e3603e074b6bb8f82aa3597210b84d4c52c34db8d6eaf506ec72159a16bb9ca8 not found: ID does not exist"
Nov 03 20:39:42.398761 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.398747    5591 scope.go:122] "RemoveContainer" containerID="b2b2ea9a9df5580537698cb2fdfb2917a47d849d546f61807ece727563d98317"
Nov 03 20:39:42.412269 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.412247    5591 scope.go:122] "RemoveContainer" containerID="b2b2ea9a9df5580537698cb2fdfb2917a47d849d546f61807ece727563d98317"
Nov 03 20:39:42.412585 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:39:42.412565    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2b2ea9a9df5580537698cb2fdfb2917a47d849d546f61807ece727563d98317\": container with ID starting with b2b2ea9a9df5580537698cb2fdfb2917a47d849d546f61807ece727563d98317 not found: ID does not exist" containerID="b2b2ea9a9df5580537698cb2fdfb2917a47d849d546f61807ece727563d98317"
Nov 03 20:39:42.412638 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.412594    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2b2ea9a9df5580537698cb2fdfb2917a47d849d546f61807ece727563d98317"} err="failed to get container status \"b2b2ea9a9df5580537698cb2fdfb2917a47d849d546f61807ece727563d98317\": rpc error: code = NotFound desc = could not find container \"b2b2ea9a9df5580537698cb2fdfb2917a47d849d546f61807ece727563d98317\": container with ID starting with b2b2ea9a9df5580537698cb2fdfb2917a47d849d546f61807ece727563d98317 not found: ID does not exist"
Nov 03 20:39:42.412638 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.412614    5591 scope.go:122] "RemoveContainer" containerID="9535fd95d2bd30b629c9e2f4735293bbbb279ec784730475cbbe7b987f192942"
Nov 03 20:39:42.417591 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.417565    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx4j4\" (UniqueName: \"kubernetes.io/projected/7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455-kube-api-access-fx4j4\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:42.417692 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.417599    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p9rh\" (UniqueName: \"kubernetes.io/projected/c9b4867c-02bd-4c72-8a46-6db1867a40c3-kube-api-access-8p9rh\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:42.424701 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.424680    5591 scope.go:122] "RemoveContainer" containerID="9535fd95d2bd30b629c9e2f4735293bbbb279ec784730475cbbe7b987f192942"
Nov 03 20:39:42.425071 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:39:42.425043    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9535fd95d2bd30b629c9e2f4735293bbbb279ec784730475cbbe7b987f192942\": container with ID starting with 9535fd95d2bd30b629c9e2f4735293bbbb279ec784730475cbbe7b987f192942 not found: ID does not exist" containerID="9535fd95d2bd30b629c9e2f4735293bbbb279ec784730475cbbe7b987f192942"
Nov 03 20:39:42.425155 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.425082    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9535fd95d2bd30b629c9e2f4735293bbbb279ec784730475cbbe7b987f192942"} err="failed to get container status \"9535fd95d2bd30b629c9e2f4735293bbbb279ec784730475cbbe7b987f192942\": rpc error: code = NotFound desc = could not find container \"9535fd95d2bd30b629c9e2f4735293bbbb279ec784730475cbbe7b987f192942\": container with ID starting with 9535fd95d2bd30b629c9e2f4735293bbbb279ec784730475cbbe7b987f192942 not found: ID does not exist"
Nov 03 20:39:42.425155 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.425107    5591 scope.go:122] "RemoveContainer" containerID="6d38f24d47ec5959d41f17a76cca7f7e4088ee4c4789f1f21a34842c36536563"
Nov 03 20:39:42.436406 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.436382    5591 scope.go:122] "RemoveContainer" containerID="6d38f24d47ec5959d41f17a76cca7f7e4088ee4c4789f1f21a34842c36536563"
Nov 03 20:39:42.436717 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:39:42.436697    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d38f24d47ec5959d41f17a76cca7f7e4088ee4c4789f1f21a34842c36536563\": container with ID starting with 6d38f24d47ec5959d41f17a76cca7f7e4088ee4c4789f1f21a34842c36536563 not found: ID does not exist" containerID="6d38f24d47ec5959d41f17a76cca7f7e4088ee4c4789f1f21a34842c36536563"
Nov 03 20:39:42.436784 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.436724    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d38f24d47ec5959d41f17a76cca7f7e4088ee4c4789f1f21a34842c36536563"} err="failed to get container status \"6d38f24d47ec5959d41f17a76cca7f7e4088ee4c4789f1f21a34842c36536563\": rpc error: code = NotFound desc = could not find container \"6d38f24d47ec5959d41f17a76cca7f7e4088ee4c4789f1f21a34842c36536563\": container with ID starting with 6d38f24d47ec5959d41f17a76cca7f7e4088ee4c4789f1f21a34842c36536563 not found: ID does not exist"
Nov 03 20:39:42.494810 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.494776    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt"]
Nov 03 20:39:42.524168 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.524135    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-f154ac-7b4cd44795-7clrt"]
Nov 03 20:39:42.542449 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.542421    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm"]
Nov 03 20:39:42.548948 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.548920    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8b7351-6494fb69f5-q99gm"]
Nov 03 20:39:42.561159 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.561119    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h"]
Nov 03 20:39:42.563847 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.563822    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h"]
Nov 03 20:39:42.577154 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.577130    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z"]
Nov 03 20:39:42.583341 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:42.583308    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f154ac-6d9544787d-zr84z"]
Nov 03 20:39:43.349704 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:43.349661    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455" path="/var/lib/kubelet/pods/7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455/volumes"
Nov 03 20:39:43.350297 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:43.350277    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9b4867c-02bd-4c72-8a46-6db1867a40c3" path="/var/lib/kubelet/pods/c9b4867c-02bd-4c72-8a46-6db1867a40c3/volumes"
Nov 03 20:39:43.350758 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:43.350737    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1031c5e-2ece-4213-ba23-33126946df5a" path="/var/lib/kubelet/pods/d1031c5e-2ece-4213-ba23-33126946df5a/volumes"
Nov 03 20:39:43.351486 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:43.351465    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe2c855d-4f89-4546-b580-e5f7e22eb9bc" path="/var/lib/kubelet/pods/fe2c855d-4f89-4546-b580-e5f7e22eb9bc/volumes"
Nov 03 20:39:44.630866 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:44.630828    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-f154ac-5c494c945d-npp5h" podUID="d1031c5e-2ece-4213-ba23-33126946df5a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.174:8443/health\": context deadline exceeded"
Nov 03 20:39:47.150901 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:47.150864    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"]
Nov 03 20:39:47.151411 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:47.151111    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh" podUID="b7189d7d-2381-4085-94b8-5b6569b7da99" containerName="istio-proxy" containerID="cri-o://1b323464aeb036e8bf2ea02ff3ea6cfd32b7b942acd495215959e915c4e0be9d" gracePeriod=30
Nov 03 20:39:47.607473 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:47.607437    5591 patch_prober.go:29] interesting pod/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:39:47.607660 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:47.607499    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh" podUID="b7189d7d-2381-4085-94b8-5b6569b7da99" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:39:47.692297 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:47.692257    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh"]
Nov 03 20:39:48.472501 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:48.472456    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:39:50.837671 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:50.837631    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2"]
Nov 03 20:39:52.460884 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.460855    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:39:52.467771 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.467740    5591 generic.go:356] "Generic (PLEG): container finished" podID="b7189d7d-2381-4085-94b8-5b6569b7da99" containerID="1b323464aeb036e8bf2ea02ff3ea6cfd32b7b942acd495215959e915c4e0be9d" exitCode=0
Nov 03 20:39:52.467918 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.467783    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"
Nov 03 20:39:52.467918 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.467824    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh" event={"ID":"b7189d7d-2381-4085-94b8-5b6569b7da99","Type":"ContainerDied","Data":"1b323464aeb036e8bf2ea02ff3ea6cfd32b7b942acd495215959e915c4e0be9d"}
Nov 03 20:39:52.467918 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.467879    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh" event={"ID":"b7189d7d-2381-4085-94b8-5b6569b7da99","Type":"ContainerDied","Data":"56e528913aeb18abc437cc93f5296433d5c5ea6ec89c35ad85dba6339be90400"}
Nov 03 20:39:52.467918 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.467905    5591 scope.go:122] "RemoveContainer" containerID="1b323464aeb036e8bf2ea02ff3ea6cfd32b7b942acd495215959e915c4e0be9d"
Nov 03 20:39:52.478374 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.478353    5591 scope.go:122] "RemoveContainer" containerID="1b323464aeb036e8bf2ea02ff3ea6cfd32b7b942acd495215959e915c4e0be9d"
Nov 03 20:39:52.478666 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:39:52.478646    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b323464aeb036e8bf2ea02ff3ea6cfd32b7b942acd495215959e915c4e0be9d\": container with ID starting with 1b323464aeb036e8bf2ea02ff3ea6cfd32b7b942acd495215959e915c4e0be9d not found: ID does not exist" containerID="1b323464aeb036e8bf2ea02ff3ea6cfd32b7b942acd495215959e915c4e0be9d"
Nov 03 20:39:52.478715 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.478679    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b323464aeb036e8bf2ea02ff3ea6cfd32b7b942acd495215959e915c4e0be9d"} err="failed to get container status \"1b323464aeb036e8bf2ea02ff3ea6cfd32b7b942acd495215959e915c4e0be9d\": rpc error: code = NotFound desc = could not find container \"1b323464aeb036e8bf2ea02ff3ea6cfd32b7b942acd495215959e915c4e0be9d\": container with ID starting with 1b323464aeb036e8bf2ea02ff3ea6cfd32b7b942acd495215959e915c4e0be9d not found: ID does not exist"
Nov 03 20:39:52.589529 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.589499    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-envoy\") pod \"b7189d7d-2381-4085-94b8-5b6569b7da99\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") "
Nov 03 20:39:52.589715 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.589607    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-token\" (UniqueName: \"kubernetes.io/projected/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-token\") pod \"b7189d7d-2381-4085-94b8-5b6569b7da99\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") "
Nov 03 20:39:52.589715 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.589648    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7189d7d-2381-4085-94b8-5b6569b7da99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7189d7d-2381-4085-94b8-5b6569b7da99-istiod-ca-cert\") pod \"b7189d7d-2381-4085-94b8-5b6569b7da99\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") "
Nov 03 20:39:52.589804 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.589715    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-workload-socket\") pod \"b7189d7d-2381-4085-94b8-5b6569b7da99\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") "
Nov 03 20:39:52.589864 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.589816    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-credential-socket\") pod \"b7189d7d-2381-4085-94b8-5b6569b7da99\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") "
Nov 03 20:39:52.589864 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.589859    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-workload-certs\") pod \"b7189d7d-2381-4085-94b8-5b6569b7da99\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") "
Nov 03 20:39:52.589986 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.589939    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-data\") pod \"b7189d7d-2381-4085-94b8-5b6569b7da99\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") "
Nov 03 20:39:52.590044 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.589991    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-podinfo\") pod \"b7189d7d-2381-4085-94b8-5b6569b7da99\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") "
Nov 03 20:39:52.590102 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.590075    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7189d7d-2381-4085-94b8-5b6569b7da99-kube-api-access-9rx64\" (UniqueName: \"kubernetes.io/projected/b7189d7d-2381-4085-94b8-5b6569b7da99-kube-api-access-9rx64\") pod \"b7189d7d-2381-4085-94b8-5b6569b7da99\" (UID: \"b7189d7d-2381-4085-94b8-5b6569b7da99\") "
Nov 03 20:39:52.590159 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.590128    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7189d7d-2381-4085-94b8-5b6569b7da99-istiod-ca-cert" pod "b7189d7d-2381-4085-94b8-5b6569b7da99" (UID: "b7189d7d-2381-4085-94b8-5b6569b7da99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:52.590363 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.590289    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-credential-socket" pod "b7189d7d-2381-4085-94b8-5b6569b7da99" (UID: "b7189d7d-2381-4085-94b8-5b6569b7da99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:52.590488 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.590325    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-data" pod "b7189d7d-2381-4085-94b8-5b6569b7da99" (UID: "b7189d7d-2381-4085-94b8-5b6569b7da99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:52.590562 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.590344    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-workload-socket" pod "b7189d7d-2381-4085-94b8-5b6569b7da99" (UID: "b7189d7d-2381-4085-94b8-5b6569b7da99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:52.590562 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.590486    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-workload-certs" pod "b7189d7d-2381-4085-94b8-5b6569b7da99" (UID: "b7189d7d-2381-4085-94b8-5b6569b7da99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:52.591052 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.590972    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:52.591052 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.590999    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:52.591052 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.591016    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:52.591052 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.591029    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7189d7d-2381-4085-94b8-5b6569b7da99-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:52.591052 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.591043    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:52.592062 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.592033    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-token" pod "b7189d7d-2381-4085-94b8-5b6569b7da99" (UID: "b7189d7d-2381-4085-94b8-5b6569b7da99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:52.592062 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.592052    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-envoy" pod "b7189d7d-2381-4085-94b8-5b6569b7da99" (UID: "b7189d7d-2381-4085-94b8-5b6569b7da99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:52.592522 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.592502    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-podinfo" pod "b7189d7d-2381-4085-94b8-5b6569b7da99" (UID: "b7189d7d-2381-4085-94b8-5b6569b7da99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:39:52.592746 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.592724    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7189d7d-2381-4085-94b8-5b6569b7da99-kube-api-access-9rx64" pod "b7189d7d-2381-4085-94b8-5b6569b7da99" (UID: "b7189d7d-2381-4085-94b8-5b6569b7da99"). InnerVolumeSpecName "kube-api-access-9rx64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:52.692597 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.692517    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:52.692597 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.692545    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rx64\" (UniqueName: \"kubernetes.io/projected/b7189d7d-2381-4085-94b8-5b6569b7da99-kube-api-access-9rx64\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:52.692597 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.692557    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:52.692597 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.692565    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7189d7d-2381-4085-94b8-5b6569b7da99-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:52.692928 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.692596    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh" podUID="190ad217-4f23-4513-be1b-405c141d460d" containerName="payload-pre-processing" containerID="cri-o://5e456c3438869f653ffb43d9ff20f40e1e1d07b0db03378218c9484b16f789a0" gracePeriod=25
Nov 03 20:39:52.791935 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.791906    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"]
Nov 03 20:39:52.796766 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.796738    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-0e4ce7-openshift-default-779c8bf98f-4bbvh"]
Nov 03 20:39:52.951827 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:52.951456    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh"
Nov 03 20:39:53.099333 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:53.099301    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/190ad217-4f23-4513-be1b-405c141d460d-kube-api-access-nwkqd\" (UniqueName: \"kubernetes.io/projected/190ad217-4f23-4513-be1b-405c141d460d-kube-api-access-nwkqd\") pod \"190ad217-4f23-4513-be1b-405c141d460d\" (UID: \"190ad217-4f23-4513-be1b-405c141d460d\") "
Nov 03 20:39:53.099549 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:53.099511    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/190ad217-4f23-4513-be1b-405c141d460d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/190ad217-4f23-4513-be1b-405c141d460d-plugins-config-volume\") pod \"190ad217-4f23-4513-be1b-405c141d460d\" (UID: \"190ad217-4f23-4513-be1b-405c141d460d\") "
Nov 03 20:39:53.099882 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:53.099857    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/190ad217-4f23-4513-be1b-405c141d460d-plugins-config-volume" pod "190ad217-4f23-4513-be1b-405c141d460d" (UID: "190ad217-4f23-4513-be1b-405c141d460d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:53.101700 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:53.101674    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190ad217-4f23-4513-be1b-405c141d460d-kube-api-access-nwkqd" pod "190ad217-4f23-4513-be1b-405c141d460d" (UID: "190ad217-4f23-4513-be1b-405c141d460d"). InnerVolumeSpecName "kube-api-access-nwkqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:53.210699 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:53.210593    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/190ad217-4f23-4513-be1b-405c141d460d-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:53.210699 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:53.210636    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwkqd\" (UniqueName: \"kubernetes.io/projected/190ad217-4f23-4513-be1b-405c141d460d-kube-api-access-nwkqd\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:53.349479 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:53.349445    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7189d7d-2381-4085-94b8-5b6569b7da99" path="/var/lib/kubelet/pods/b7189d7d-2381-4085-94b8-5b6569b7da99/volumes"
Nov 03 20:39:53.477442 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:53.477348    5591 generic.go:356] "Generic (PLEG): container finished" podID="190ad217-4f23-4513-be1b-405c141d460d" containerID="5e456c3438869f653ffb43d9ff20f40e1e1d07b0db03378218c9484b16f789a0" exitCode=0
Nov 03 20:39:53.477442 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:53.477397    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh"
Nov 03 20:39:53.477951 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:53.477431    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh" event={"ID":"190ad217-4f23-4513-be1b-405c141d460d","Type":"ContainerDied","Data":"5e456c3438869f653ffb43d9ff20f40e1e1d07b0db03378218c9484b16f789a0"}
Nov 03 20:39:53.477951 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:53.477490    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh" event={"ID":"190ad217-4f23-4513-be1b-405c141d460d","Type":"ContainerDied","Data":"86aa1dc3b379ca4fb8046ebb7f4b173a0d4929622b3496544e0e25240705b8c9"}
Nov 03 20:39:53.477951 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:53.477514    5591 scope.go:122] "RemoveContainer" containerID="5e456c3438869f653ffb43d9ff20f40e1e1d07b0db03378218c9484b16f789a0"
Nov 03 20:39:53.487411 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:53.487395    5591 scope.go:122] "RemoveContainer" containerID="5e456c3438869f653ffb43d9ff20f40e1e1d07b0db03378218c9484b16f789a0"
Nov 03 20:39:53.487703 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:39:53.487682    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e456c3438869f653ffb43d9ff20f40e1e1d07b0db03378218c9484b16f789a0\": container with ID starting with 5e456c3438869f653ffb43d9ff20f40e1e1d07b0db03378218c9484b16f789a0 not found: ID does not exist" containerID="5e456c3438869f653ffb43d9ff20f40e1e1d07b0db03378218c9484b16f789a0"
Nov 03 20:39:53.487776 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:53.487710    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e456c3438869f653ffb43d9ff20f40e1e1d07b0db03378218c9484b16f789a0"} err="failed to get container status \"5e456c3438869f653ffb43d9ff20f40e1e1d07b0db03378218c9484b16f789a0\": rpc error: code = NotFound desc = could not find container \"5e456c3438869f653ffb43d9ff20f40e1e1d07b0db03378218c9484b16f789a0\": container with ID starting with 5e456c3438869f653ffb43d9ff20f40e1e1d07b0db03378218c9484b16f789a0 not found: ID does not exist"
Nov 03 20:39:53.498915 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:53.498889    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh"]
Nov 03 20:39:53.503919 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:53.503898    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-2b5703-869468bbc5-546nh"]
Nov 03 20:39:54.003949 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:54.003909    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp"]
Nov 03 20:39:54.020215 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:54.020167    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v"]
Nov 03 20:39:55.314278 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:55.314242    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq"]
Nov 03 20:39:55.315111 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:55.315052    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq" podUID="22691b0c-82ca-4d24-85e6-fc861fcb73c0" containerName="maas-api" containerID="cri-o://94fc9a9e588fa50991d89ce93e91e187ec4681ebc1b61ed279c3f9e9df668dd5" gracePeriod=30
Nov 03 20:39:55.351884 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:55.351849    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="190ad217-4f23-4513-be1b-405c141d460d" path="/var/lib/kubelet/pods/190ad217-4f23-4513-be1b-405c141d460d/volumes"
Nov 03 20:39:55.493773 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:55.493606    5591 generic.go:356] "Generic (PLEG): container finished" podID="22691b0c-82ca-4d24-85e6-fc861fcb73c0" containerID="94fc9a9e588fa50991d89ce93e91e187ec4681ebc1b61ed279c3f9e9df668dd5" exitCode=0
Nov 03 20:39:55.493773 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:55.493630    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq" event={"ID":"22691b0c-82ca-4d24-85e6-fc861fcb73c0","Type":"ContainerDied","Data":"94fc9a9e588fa50991d89ce93e91e187ec4681ebc1b61ed279c3f9e9df668dd5"}
Nov 03 20:39:55.678989 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:55.678922    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq"
Nov 03 20:39:55.837962 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:55.837925    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2" podUID="9d9d77f5-8819-447e-b0a0-c27ce2d1d498" containerName="payload-pre-processing" containerID="cri-o://42fc421eb86b99bbda2dbb18c1f0eaf5633c92dd78fbe4ed59273596eaaf14c7" gracePeriod=25
Nov 03 20:39:55.851544 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:55.851517    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22691b0c-82ca-4d24-85e6-fc861fcb73c0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22691b0c-82ca-4d24-85e6-fc861fcb73c0-openshift-service-ca\") pod \"22691b0c-82ca-4d24-85e6-fc861fcb73c0\" (UID: \"22691b0c-82ca-4d24-85e6-fc861fcb73c0\") "
Nov 03 20:39:55.851693 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:55.851639    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22691b0c-82ca-4d24-85e6-fc861fcb73c0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22691b0c-82ca-4d24-85e6-fc861fcb73c0-maas-api-tls\") pod \"22691b0c-82ca-4d24-85e6-fc861fcb73c0\" (UID: \"22691b0c-82ca-4d24-85e6-fc861fcb73c0\") "
Nov 03 20:39:55.851693 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:55.851677    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22691b0c-82ca-4d24-85e6-fc861fcb73c0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/22691b0c-82ca-4d24-85e6-fc861fcb73c0-metrics-certs\") pod \"22691b0c-82ca-4d24-85e6-fc861fcb73c0\" (UID: \"22691b0c-82ca-4d24-85e6-fc861fcb73c0\") "
Nov 03 20:39:55.851787 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:55.851751    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22691b0c-82ca-4d24-85e6-fc861fcb73c0-kube-api-access-8bqvv\" (UniqueName: \"kubernetes.io/projected/22691b0c-82ca-4d24-85e6-fc861fcb73c0-kube-api-access-8bqvv\") pod \"22691b0c-82ca-4d24-85e6-fc861fcb73c0\" (UID: \"22691b0c-82ca-4d24-85e6-fc861fcb73c0\") "
Nov 03 20:39:55.851885 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:55.851864    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22691b0c-82ca-4d24-85e6-fc861fcb73c0-openshift-service-ca" pod "22691b0c-82ca-4d24-85e6-fc861fcb73c0" (UID: "22691b0c-82ca-4d24-85e6-fc861fcb73c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:55.852357 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:55.852324    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22691b0c-82ca-4d24-85e6-fc861fcb73c0-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:55.859447 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:55.859379    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22691b0c-82ca-4d24-85e6-fc861fcb73c0-metrics-certs" pod "22691b0c-82ca-4d24-85e6-fc861fcb73c0" (UID: "22691b0c-82ca-4d24-85e6-fc861fcb73c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:55.859447 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:55.859394    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22691b0c-82ca-4d24-85e6-fc861fcb73c0-maas-api-tls" pod "22691b0c-82ca-4d24-85e6-fc861fcb73c0" (UID: "22691b0c-82ca-4d24-85e6-fc861fcb73c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:55.859447 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:55.859431    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22691b0c-82ca-4d24-85e6-fc861fcb73c0-kube-api-access-8bqvv" pod "22691b0c-82ca-4d24-85e6-fc861fcb73c0" (UID: "22691b0c-82ca-4d24-85e6-fc861fcb73c0"). InnerVolumeSpecName "kube-api-access-8bqvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:55.953883 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:55.953855    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22691b0c-82ca-4d24-85e6-fc861fcb73c0-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:55.953985 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:55.953885    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22691b0c-82ca-4d24-85e6-fc861fcb73c0-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:55.953985 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:55.953896    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bqvv\" (UniqueName: \"kubernetes.io/projected/22691b0c-82ca-4d24-85e6-fc861fcb73c0-kube-api-access-8bqvv\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:56.088552 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.088525    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2"
Nov 03 20:39:56.160713 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.160679    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n"]
Nov 03 20:39:56.160983 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.160929    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n" podUID="9d7c8f87-8009-4efa-a92f-3dd314f7bac0" containerName="maas-api" containerID="cri-o://72aa9d5a68cb141b51757eed21f00436a942681092448f149bf2bdec38a2e8f7" gracePeriod=30
Nov 03 20:39:56.258847 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.258762    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d9d77f5-8819-447e-b0a0-c27ce2d1d498-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d9d77f5-8819-447e-b0a0-c27ce2d1d498-plugins-config-volume\") pod \"9d9d77f5-8819-447e-b0a0-c27ce2d1d498\" (UID: \"9d9d77f5-8819-447e-b0a0-c27ce2d1d498\") "
Nov 03 20:39:56.258847 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.258833    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d9d77f5-8819-447e-b0a0-c27ce2d1d498-kube-api-access-slj5q\" (UniqueName: \"kubernetes.io/projected/9d9d77f5-8819-447e-b0a0-c27ce2d1d498-kube-api-access-slj5q\") pod \"9d9d77f5-8819-447e-b0a0-c27ce2d1d498\" (UID: \"9d9d77f5-8819-447e-b0a0-c27ce2d1d498\") "
Nov 03 20:39:56.259132 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.259105    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d9d77f5-8819-447e-b0a0-c27ce2d1d498-plugins-config-volume" pod "9d9d77f5-8819-447e-b0a0-c27ce2d1d498" (UID: "9d9d77f5-8819-447e-b0a0-c27ce2d1d498"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:56.259509 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.259487    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d9d77f5-8819-447e-b0a0-c27ce2d1d498-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:56.261124 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.261101    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d9d77f5-8819-447e-b0a0-c27ce2d1d498-kube-api-access-slj5q" pod "9d9d77f5-8819-447e-b0a0-c27ce2d1d498" (UID: "9d9d77f5-8819-447e-b0a0-c27ce2d1d498"). InnerVolumeSpecName "kube-api-access-slj5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:56.432189 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.430790    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slj5q\" (UniqueName: \"kubernetes.io/projected/9d9d77f5-8819-447e-b0a0-c27ce2d1d498-kube-api-access-slj5q\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:56.505339 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.505303    5591 generic.go:356] "Generic (PLEG): container finished" podID="9d9d77f5-8819-447e-b0a0-c27ce2d1d498" containerID="42fc421eb86b99bbda2dbb18c1f0eaf5633c92dd78fbe4ed59273596eaaf14c7" exitCode=0
Nov 03 20:39:56.506127 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.505566    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2"
Nov 03 20:39:56.506288 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.505606    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2" event={"ID":"9d9d77f5-8819-447e-b0a0-c27ce2d1d498","Type":"ContainerDied","Data":"42fc421eb86b99bbda2dbb18c1f0eaf5633c92dd78fbe4ed59273596eaaf14c7"}
Nov 03 20:39:56.506401 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.506350    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2" event={"ID":"9d9d77f5-8819-447e-b0a0-c27ce2d1d498","Type":"ContainerDied","Data":"e170adf957110c32bf88c376395c8fac7913e4494ee447750731ba3cbabad5f9"}
Nov 03 20:39:56.506401 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.506385    5591 scope.go:122] "RemoveContainer" containerID="42fc421eb86b99bbda2dbb18c1f0eaf5633c92dd78fbe4ed59273596eaaf14c7"
Nov 03 20:39:56.508554 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.508518    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq" event={"ID":"22691b0c-82ca-4d24-85e6-fc861fcb73c0","Type":"ContainerDied","Data":"650fea909e0e9e2de2d0134791ee81285902cfff4098178680bec7aa3034fbc0"}
Nov 03 20:39:56.508692 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.508608    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq"
Nov 03 20:39:56.512811 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.512783    5591 generic.go:356] "Generic (PLEG): container finished" podID="9d7c8f87-8009-4efa-a92f-3dd314f7bac0" containerID="72aa9d5a68cb141b51757eed21f00436a942681092448f149bf2bdec38a2e8f7" exitCode=0
Nov 03 20:39:56.512907 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.512836    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n" event={"ID":"9d7c8f87-8009-4efa-a92f-3dd314f7bac0","Type":"ContainerDied","Data":"72aa9d5a68cb141b51757eed21f00436a942681092448f149bf2bdec38a2e8f7"}
Nov 03 20:39:56.512907 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.512881    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n" event={"ID":"9d7c8f87-8009-4efa-a92f-3dd314f7bac0","Type":"ContainerDied","Data":"afbb114bebb59384e355d1e8c4b969d0957b5a80d2ce5a73096c8c7859c25632"}
Nov 03 20:39:56.513007 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.512906    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afbb114bebb59384e355d1e8c4b969d0957b5a80d2ce5a73096c8c7859c25632"
Nov 03 20:39:56.532901 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.532880    5591 scope.go:122] "RemoveContainer" containerID="42fc421eb86b99bbda2dbb18c1f0eaf5633c92dd78fbe4ed59273596eaaf14c7"
Nov 03 20:39:56.533176 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:39:56.533157    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42fc421eb86b99bbda2dbb18c1f0eaf5633c92dd78fbe4ed59273596eaaf14c7\": container with ID starting with 42fc421eb86b99bbda2dbb18c1f0eaf5633c92dd78fbe4ed59273596eaaf14c7 not found: ID does not exist" containerID="42fc421eb86b99bbda2dbb18c1f0eaf5633c92dd78fbe4ed59273596eaaf14c7"
Nov 03 20:39:56.533294 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.533186    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42fc421eb86b99bbda2dbb18c1f0eaf5633c92dd78fbe4ed59273596eaaf14c7"} err="failed to get container status \"42fc421eb86b99bbda2dbb18c1f0eaf5633c92dd78fbe4ed59273596eaaf14c7\": rpc error: code = NotFound desc = could not find container \"42fc421eb86b99bbda2dbb18c1f0eaf5633c92dd78fbe4ed59273596eaaf14c7\": container with ID starting with 42fc421eb86b99bbda2dbb18c1f0eaf5633c92dd78fbe4ed59273596eaaf14c7 not found: ID does not exist"
Nov 03 20:39:56.533294 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.533254    5591 scope.go:122] "RemoveContainer" containerID="94fc9a9e588fa50991d89ce93e91e187ec4681ebc1b61ed279c3f9e9df668dd5"
Nov 03 20:39:56.535371 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.535354    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n"
Nov 03 20:39:56.548119 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.548097    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq"]
Nov 03 20:39:56.554133 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.554111    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-43c674-56c4f8ffb8-q5bgq"]
Nov 03 20:39:56.567352 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.567328    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2"]
Nov 03 20:39:56.571371 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.571348    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-43c674-66c7dd866c-dl2t2"]
Nov 03 20:39:56.632716 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.632678    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-metrics-certs\") pod \"9d7c8f87-8009-4efa-a92f-3dd314f7bac0\" (UID: \"9d7c8f87-8009-4efa-a92f-3dd314f7bac0\") "
Nov 03 20:39:56.632911 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.632747    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-openshift-service-ca\") pod \"9d7c8f87-8009-4efa-a92f-3dd314f7bac0\" (UID: \"9d7c8f87-8009-4efa-a92f-3dd314f7bac0\") "
Nov 03 20:39:56.633005 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.632931    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-maas-api-tls\") pod \"9d7c8f87-8009-4efa-a92f-3dd314f7bac0\" (UID: \"9d7c8f87-8009-4efa-a92f-3dd314f7bac0\") "
Nov 03 20:39:56.633421 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.633106    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-openshift-service-ca" pod "9d7c8f87-8009-4efa-a92f-3dd314f7bac0" (UID: "9d7c8f87-8009-4efa-a92f-3dd314f7bac0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:56.633421 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.633398    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:56.635152 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.635125    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-metrics-certs" pod "9d7c8f87-8009-4efa-a92f-3dd314f7bac0" (UID: "9d7c8f87-8009-4efa-a92f-3dd314f7bac0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:56.635433 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.635411    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-maas-api-tls" pod "9d7c8f87-8009-4efa-a92f-3dd314f7bac0" (UID: "9d7c8f87-8009-4efa-a92f-3dd314f7bac0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:56.659765 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.659734    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml" podUID="57c85ce9-ebbf-4c18-b40c-e09ea4e9c068" containerName="main" containerID="cri-o://5bb6854de3819ef662794d84b6fb0c8de6ed539fbb85cbef337ccd2b0aa7fcf8" gracePeriod=45
Nov 03 20:39:56.671684 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.671638    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" containerID="cri-o://debd15accce6666c5ffb2b5dacffb9f4a02cc4706e520987ed1a0fa8c7002839" gracePeriod=45
Nov 03 20:39:56.719445 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.719413    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml" podUID="57c85ce9-ebbf-4c18-b40c-e09ea4e9c068" containerName="main" probeResult="failure" output="Get \"https://10.134.1.12:8000/ready\": dial tcp 10.134.1.12:8000: connect: connection refused"
Nov 03 20:39:56.734998 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.734971    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-kube-api-access-pz2gw\" (UniqueName: \"kubernetes.io/projected/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-kube-api-access-pz2gw\") pod \"9d7c8f87-8009-4efa-a92f-3dd314f7bac0\" (UID: \"9d7c8f87-8009-4efa-a92f-3dd314f7bac0\") "
Nov 03 20:39:56.735411 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.735398    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:56.735483 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.735413    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:56.737322 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.737295    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-kube-api-access-pz2gw" pod "9d7c8f87-8009-4efa-a92f-3dd314f7bac0" (UID: "9d7c8f87-8009-4efa-a92f-3dd314f7bac0"). InnerVolumeSpecName "kube-api-access-pz2gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:56.795243 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.795163    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": dial tcp 10.134.0.59:8000: connect: connection refused"
Nov 03 20:39:56.836890 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.836594    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz2gw\" (UniqueName: \"kubernetes.io/projected/9d7c8f87-8009-4efa-a92f-3dd314f7bac0-kube-api-access-pz2gw\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:56.930315 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.930276    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"
Nov 03 20:39:56.938011 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.937985    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-model-cache\") pod \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\" (UID: \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\") "
Nov 03 20:39:56.938176 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.938076    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-tmp-dir\") pod \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\" (UID: \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\") "
Nov 03 20:39:56.938176 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.938107    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-home\" (UniqueName: \"kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-home\") pod \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\" (UID: \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\") "
Nov 03 20:39:56.938330 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.938240    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-tls-certs\" (UniqueName: \"kubernetes.io/secret/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-tls-certs\") pod \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\" (UID: \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\") "
Nov 03 20:39:56.938393 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.938370    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-kube-api-access-xd6cq\" (UniqueName: \"kubernetes.io/projected/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-kube-api-access-xd6cq\") pod \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\" (UID: \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\") "
Nov 03 20:39:56.938517 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.938491    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-home" pod "57c85ce9-ebbf-4c18-b40c-e09ea4e9c068" (UID: "57c85ce9-ebbf-4c18-b40c-e09ea4e9c068"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:56.938517 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.938504    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-dshm\" (UniqueName: \"kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-dshm\") pod \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\" (UID: \"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068\") "
Nov 03 20:39:56.938754 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.938731    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-tmp-dir" pod "57c85ce9-ebbf-4c18-b40c-e09ea4e9c068" (UID: "57c85ce9-ebbf-4c18-b40c-e09ea4e9c068"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:56.938850 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.938831    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-model-cache" pod "57c85ce9-ebbf-4c18-b40c-e09ea4e9c068" (UID: "57c85ce9-ebbf-4c18-b40c-e09ea4e9c068"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:56.939308 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.939238    5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-model-cache\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:56.939428 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.939311    5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-tmp-dir\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:56.939428 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.939327    5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-home\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:56.941535 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.941507    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-tls-certs" pod "57c85ce9-ebbf-4c18-b40c-e09ea4e9c068" (UID: "57c85ce9-ebbf-4c18-b40c-e09ea4e9c068"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:56.941630 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.941565    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-dshm" pod "57c85ce9-ebbf-4c18-b40c-e09ea4e9c068" (UID: "57c85ce9-ebbf-4c18-b40c-e09ea4e9c068"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:56.941873 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.941849    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-kube-api-access-xd6cq" pod "57c85ce9-ebbf-4c18-b40c-e09ea4e9c068" (UID: "57c85ce9-ebbf-4c18-b40c-e09ea4e9c068"). InnerVolumeSpecName "kube-api-access-xd6cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:56.950548 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:56.950015    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 20:39:57.040437 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.040343    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58f1605e-5db2-4a8d-9b62-4c84e113ac33-tls-certs\" (UniqueName: \"kubernetes.io/secret/58f1605e-5db2-4a8d-9b62-4c84e113ac33-tls-certs\") pod \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\" (UID: \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\") "
Nov 03 20:39:57.040607 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.040454    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-dshm\" (UniqueName: \"kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-dshm\") pod \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\" (UID: \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\") "
Nov 03 20:39:57.040607 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.040504    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58f1605e-5db2-4a8d-9b62-4c84e113ac33-kube-api-access-h78gf\" (UniqueName: \"kubernetes.io/projected/58f1605e-5db2-4a8d-9b62-4c84e113ac33-kube-api-access-h78gf\") pod \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\" (UID: \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\") "
Nov 03 20:39:57.040607 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.040541    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-home\" (UniqueName: \"kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-home\") pod \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\" (UID: \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\") "
Nov 03 20:39:57.040829 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.040609    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-tmp-dir\") pod \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\" (UID: \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\") "
Nov 03 20:39:57.040829 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.040694    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-model-cache\") pod \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\" (UID: \"58f1605e-5db2-4a8d-9b62-4c84e113ac33\") "
Nov 03 20:39:57.040929 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.040867    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-home" pod "58f1605e-5db2-4a8d-9b62-4c84e113ac33" (UID: "58f1605e-5db2-4a8d-9b62-4c84e113ac33"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:57.041026 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.040993    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-tmp-dir" pod "58f1605e-5db2-4a8d-9b62-4c84e113ac33" (UID: "58f1605e-5db2-4a8d-9b62-4c84e113ac33"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:57.041110 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.041016    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-model-cache" pod "58f1605e-5db2-4a8d-9b62-4c84e113ac33" (UID: "58f1605e-5db2-4a8d-9b62-4c84e113ac33"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:57.041534 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.041511    5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-tmp-dir\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:57.041602 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.041546    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd6cq\" (UniqueName: \"kubernetes.io/projected/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-kube-api-access-xd6cq\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:57.041602 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.041564    5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-model-cache\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:57.041602 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.041580    5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-dshm\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:57.041602 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.041593    5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068-tls-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:57.041748 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.041607    5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-home\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:57.042893 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.042869    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58f1605e-5db2-4a8d-9b62-4c84e113ac33-tls-certs" pod "58f1605e-5db2-4a8d-9b62-4c84e113ac33" (UID: "58f1605e-5db2-4a8d-9b62-4c84e113ac33"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:39:57.043297 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.043268    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-dshm" pod "58f1605e-5db2-4a8d-9b62-4c84e113ac33" (UID: "58f1605e-5db2-4a8d-9b62-4c84e113ac33"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:39:57.043378 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.043324    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f1605e-5db2-4a8d-9b62-4c84e113ac33-kube-api-access-h78gf" pod "58f1605e-5db2-4a8d-9b62-4c84e113ac33" (UID: "58f1605e-5db2-4a8d-9b62-4c84e113ac33"). InnerVolumeSpecName "kube-api-access-h78gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:57.142946 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.142915    5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58f1605e-5db2-4a8d-9b62-4c84e113ac33-dshm\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:57.142946 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.142944    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h78gf\" (UniqueName: \"kubernetes.io/projected/58f1605e-5db2-4a8d-9b62-4c84e113ac33-kube-api-access-h78gf\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:57.142946 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.142954    5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58f1605e-5db2-4a8d-9b62-4c84e113ac33-tls-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:57.376641 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.376602    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22691b0c-82ca-4d24-85e6-fc861fcb73c0" path="/var/lib/kubelet/pods/22691b0c-82ca-4d24-85e6-fc861fcb73c0/volumes"
Nov 03 20:39:57.377146 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.377123    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d9d77f5-8819-447e-b0a0-c27ce2d1d498" path="/var/lib/kubelet/pods/9d9d77f5-8819-447e-b0a0-c27ce2d1d498/volumes"
Nov 03 20:39:57.520410 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.520377    5591 generic.go:356] "Generic (PLEG): container finished" podID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerID="debd15accce6666c5ffb2b5dacffb9f4a02cc4706e520987ed1a0fa8c7002839" exitCode=0
Nov 03 20:39:57.520875 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.520442    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"
Nov 03 20:39:57.520875 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.520445    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" event={"ID":"58f1605e-5db2-4a8d-9b62-4c84e113ac33","Type":"ContainerDied","Data":"debd15accce6666c5ffb2b5dacffb9f4a02cc4706e520987ed1a0fa8c7002839"}
Nov 03 20:39:57.520875 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.520486    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz" event={"ID":"58f1605e-5db2-4a8d-9b62-4c84e113ac33","Type":"ContainerDied","Data":"6d15c4e829b760a48de4f49a570fcd27e527c556856bb45fcb0d6086e937700d"}
Nov 03 20:39:57.520875 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.520502    5591 scope.go:122] "RemoveContainer" containerID="debd15accce6666c5ffb2b5dacffb9f4a02cc4706e520987ed1a0fa8c7002839"
Nov 03 20:39:57.523639 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.523617    5591 generic.go:356] "Generic (PLEG): container finished" podID="57c85ce9-ebbf-4c18-b40c-e09ea4e9c068" containerID="5bb6854de3819ef662794d84b6fb0c8de6ed539fbb85cbef337ccd2b0aa7fcf8" exitCode=0
Nov 03 20:39:57.523762 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.523662    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"
Nov 03 20:39:57.523762 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.523708    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml" event={"ID":"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068","Type":"ContainerDied","Data":"5bb6854de3819ef662794d84b6fb0c8de6ed539fbb85cbef337ccd2b0aa7fcf8"}
Nov 03 20:39:57.523762 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.523747    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml" event={"ID":"57c85ce9-ebbf-4c18-b40c-e09ea4e9c068","Type":"ContainerDied","Data":"cf95d59efa66e5ce5464406e2b0de99225bfacc139eadb17365a156f2e72e620"}
Nov 03 20:39:57.524985 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.524964    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n"
Nov 03 20:39:57.531714 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.531692    5591 scope.go:122] "RemoveContainer" containerID="debd15accce6666c5ffb2b5dacffb9f4a02cc4706e520987ed1a0fa8c7002839"
Nov 03 20:39:57.532018 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:39:57.531995    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"debd15accce6666c5ffb2b5dacffb9f4a02cc4706e520987ed1a0fa8c7002839\": container with ID starting with debd15accce6666c5ffb2b5dacffb9f4a02cc4706e520987ed1a0fa8c7002839 not found: ID does not exist" containerID="debd15accce6666c5ffb2b5dacffb9f4a02cc4706e520987ed1a0fa8c7002839"
Nov 03 20:39:57.532127 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.532027    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"debd15accce6666c5ffb2b5dacffb9f4a02cc4706e520987ed1a0fa8c7002839"} err="failed to get container status \"debd15accce6666c5ffb2b5dacffb9f4a02cc4706e520987ed1a0fa8c7002839\": rpc error: code = NotFound desc = could not find container \"debd15accce6666c5ffb2b5dacffb9f4a02cc4706e520987ed1a0fa8c7002839\": container with ID starting with debd15accce6666c5ffb2b5dacffb9f4a02cc4706e520987ed1a0fa8c7002839 not found: ID does not exist"
Nov 03 20:39:57.532127 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.532048    5591 scope.go:122] "RemoveContainer" containerID="5bb6854de3819ef662794d84b6fb0c8de6ed539fbb85cbef337ccd2b0aa7fcf8"
Nov 03 20:39:57.540218 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.540178    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"]
Nov 03 20:39:57.541841 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.541819    5591 scope.go:122] "RemoveContainer" containerID="5bb6854de3819ef662794d84b6fb0c8de6ed539fbb85cbef337ccd2b0aa7fcf8"
Nov 03 20:39:57.542415 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:39:57.542388    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bb6854de3819ef662794d84b6fb0c8de6ed539fbb85cbef337ccd2b0aa7fcf8\": container with ID starting with 5bb6854de3819ef662794d84b6fb0c8de6ed539fbb85cbef337ccd2b0aa7fcf8 not found: ID does not exist" containerID="5bb6854de3819ef662794d84b6fb0c8de6ed539fbb85cbef337ccd2b0aa7fcf8"
Nov 03 20:39:57.542496 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.542424    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bb6854de3819ef662794d84b6fb0c8de6ed539fbb85cbef337ccd2b0aa7fcf8"} err="failed to get container status \"5bb6854de3819ef662794d84b6fb0c8de6ed539fbb85cbef337ccd2b0aa7fcf8\": rpc error: code = NotFound desc = could not find container \"5bb6854de3819ef662794d84b6fb0c8de6ed539fbb85cbef337ccd2b0aa7fcf8\": container with ID starting with 5bb6854de3819ef662794d84b6fb0c8de6ed539fbb85cbef337ccd2b0aa7fcf8 not found: ID does not exist"
Nov 03 20:39:57.544902 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.544882    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/premium-w2-2b5703-kserve-69744b97b7-m68tz"]
Nov 03 20:39:57.559865 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.559834    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"]
Nov 03 20:39:57.566244 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.566196    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2b5703/distinct-w2-2b5703-kserve-676bd58bd6-fw6ml"]
Nov 03 20:39:57.579911 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.579872    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n"]
Nov 03 20:39:57.583478 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:57.583455    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-bfc240-6c45db4477-dpb4n"]
Nov 03 20:39:58.729423 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:39:58.729385    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:39:59.004905 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.004789    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp" podUID="e3c18563-3261-4841-81d2-3db2ec91abd3" containerName="payload-processing" containerID="cri-o://c14f54124d5b880b893e9a8f2e0cea84cd77188b3020d6fe069fbc712f21aeec" gracePeriod=25
Nov 03 20:39:59.021352 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.021315    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v" podUID="26612e45-5a14-497a-9e90-06f336b9d0d1" containerName="payload-pre-processing" containerID="cri-o://abfd735653fa89b6e2fdf65fd9115e22269f7a36ae139c6e7f1975bbf90f1a3a" gracePeriod=25
Nov 03 20:39:59.331470 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.331441    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp"
Nov 03 20:39:59.348634 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.348609    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v"
Nov 03 20:39:59.350929 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.350891    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57c85ce9-ebbf-4c18-b40c-e09ea4e9c068" path="/var/lib/kubelet/pods/57c85ce9-ebbf-4c18-b40c-e09ea4e9c068/volumes"
Nov 03 20:39:59.351495 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.351475    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" path="/var/lib/kubelet/pods/58f1605e-5db2-4a8d-9b62-4c84e113ac33/volumes"
Nov 03 20:39:59.352060 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.352041    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d7c8f87-8009-4efa-a92f-3dd314f7bac0" path="/var/lib/kubelet/pods/9d7c8f87-8009-4efa-a92f-3dd314f7bac0/volumes"
Nov 03 20:39:59.394028 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.393987    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26612e45-5a14-497a-9e90-06f336b9d0d1-kube-api-access-p4mqv\" (UniqueName: \"kubernetes.io/projected/26612e45-5a14-497a-9e90-06f336b9d0d1-kube-api-access-p4mqv\") pod \"26612e45-5a14-497a-9e90-06f336b9d0d1\" (UID: \"26612e45-5a14-497a-9e90-06f336b9d0d1\") "
Nov 03 20:39:59.394258 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.394131    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26612e45-5a14-497a-9e90-06f336b9d0d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26612e45-5a14-497a-9e90-06f336b9d0d1-plugins-config-volume\") pod \"26612e45-5a14-497a-9e90-06f336b9d0d1\" (UID: \"26612e45-5a14-497a-9e90-06f336b9d0d1\") "
Nov 03 20:39:59.394258 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.394185    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3c18563-3261-4841-81d2-3db2ec91abd3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c18563-3261-4841-81d2-3db2ec91abd3-plugins-config-volume\") pod \"e3c18563-3261-4841-81d2-3db2ec91abd3\" (UID: \"e3c18563-3261-4841-81d2-3db2ec91abd3\") "
Nov 03 20:39:59.394398 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.394261    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c18563-3261-4841-81d2-3db2ec91abd3-kube-api-access-fqj6c\" (UniqueName: \"kubernetes.io/projected/e3c18563-3261-4841-81d2-3db2ec91abd3-kube-api-access-fqj6c\") pod \"e3c18563-3261-4841-81d2-3db2ec91abd3\" (UID: \"e3c18563-3261-4841-81d2-3db2ec91abd3\") "
Nov 03 20:39:59.394544 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.394516    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26612e45-5a14-497a-9e90-06f336b9d0d1-plugins-config-volume" pod "26612e45-5a14-497a-9e90-06f336b9d0d1" (UID: "26612e45-5a14-497a-9e90-06f336b9d0d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:59.394928 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.394906    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26612e45-5a14-497a-9e90-06f336b9d0d1-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:59.395012 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.394914    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3c18563-3261-4841-81d2-3db2ec91abd3-plugins-config-volume" pod "e3c18563-3261-4841-81d2-3db2ec91abd3" (UID: "e3c18563-3261-4841-81d2-3db2ec91abd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:39:59.397668 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.397631    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c18563-3261-4841-81d2-3db2ec91abd3-kube-api-access-fqj6c" pod "e3c18563-3261-4841-81d2-3db2ec91abd3" (UID: "e3c18563-3261-4841-81d2-3db2ec91abd3"). InnerVolumeSpecName "kube-api-access-fqj6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:59.397828 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.397740    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26612e45-5a14-497a-9e90-06f336b9d0d1-kube-api-access-p4mqv" pod "26612e45-5a14-497a-9e90-06f336b9d0d1" (UID: "26612e45-5a14-497a-9e90-06f336b9d0d1"). InnerVolumeSpecName "kube-api-access-p4mqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:39:59.495955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.495908    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqj6c\" (UniqueName: \"kubernetes.io/projected/e3c18563-3261-4841-81d2-3db2ec91abd3-kube-api-access-fqj6c\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:59.495955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.495942    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4mqv\" (UniqueName: \"kubernetes.io/projected/26612e45-5a14-497a-9e90-06f336b9d0d1-kube-api-access-p4mqv\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:59.495955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.495953    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c18563-3261-4841-81d2-3db2ec91abd3-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:39:59.544720 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.544688    5591 generic.go:356] "Generic (PLEG): container finished" podID="26612e45-5a14-497a-9e90-06f336b9d0d1" containerID="abfd735653fa89b6e2fdf65fd9115e22269f7a36ae139c6e7f1975bbf90f1a3a" exitCode=0
Nov 03 20:39:59.544947 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.544755    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v"
Nov 03 20:39:59.544947 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.544784    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v" event={"ID":"26612e45-5a14-497a-9e90-06f336b9d0d1","Type":"ContainerDied","Data":"abfd735653fa89b6e2fdf65fd9115e22269f7a36ae139c6e7f1975bbf90f1a3a"}
Nov 03 20:39:59.544947 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.544839    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v" event={"ID":"26612e45-5a14-497a-9e90-06f336b9d0d1","Type":"ContainerDied","Data":"0471b0574acde8b43f180b30010cc163e72a928336aa5f456a91aca020f7da78"}
Nov 03 20:39:59.544947 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.544870    5591 scope.go:122] "RemoveContainer" containerID="abfd735653fa89b6e2fdf65fd9115e22269f7a36ae139c6e7f1975bbf90f1a3a"
Nov 03 20:39:59.546473 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.546454    5591 generic.go:356] "Generic (PLEG): container finished" podID="e3c18563-3261-4841-81d2-3db2ec91abd3" containerID="c14f54124d5b880b893e9a8f2e0cea84cd77188b3020d6fe069fbc712f21aeec" exitCode=0
Nov 03 20:39:59.546577 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.546500    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp" event={"ID":"e3c18563-3261-4841-81d2-3db2ec91abd3","Type":"ContainerDied","Data":"c14f54124d5b880b893e9a8f2e0cea84cd77188b3020d6fe069fbc712f21aeec"}
Nov 03 20:39:59.546577 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.546532    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp" event={"ID":"e3c18563-3261-4841-81d2-3db2ec91abd3","Type":"ContainerDied","Data":"f1605e8af12a8b828cc2ce9a4bfaa5f80b705a0546921c679861f060d42076a3"}
Nov 03 20:39:59.546728 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.546587    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp"
Nov 03 20:39:59.556941 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.556920    5591 scope.go:122] "RemoveContainer" containerID="abfd735653fa89b6e2fdf65fd9115e22269f7a36ae139c6e7f1975bbf90f1a3a"
Nov 03 20:39:59.557380 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:39:59.557356    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abfd735653fa89b6e2fdf65fd9115e22269f7a36ae139c6e7f1975bbf90f1a3a\": container with ID starting with abfd735653fa89b6e2fdf65fd9115e22269f7a36ae139c6e7f1975bbf90f1a3a not found: ID does not exist" containerID="abfd735653fa89b6e2fdf65fd9115e22269f7a36ae139c6e7f1975bbf90f1a3a"
Nov 03 20:39:59.557481 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.557393    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abfd735653fa89b6e2fdf65fd9115e22269f7a36ae139c6e7f1975bbf90f1a3a"} err="failed to get container status \"abfd735653fa89b6e2fdf65fd9115e22269f7a36ae139c6e7f1975bbf90f1a3a\": rpc error: code = NotFound desc = could not find container \"abfd735653fa89b6e2fdf65fd9115e22269f7a36ae139c6e7f1975bbf90f1a3a\": container with ID starting with abfd735653fa89b6e2fdf65fd9115e22269f7a36ae139c6e7f1975bbf90f1a3a not found: ID does not exist"
Nov 03 20:39:59.557481 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.557415    5591 scope.go:122] "RemoveContainer" containerID="c14f54124d5b880b893e9a8f2e0cea84cd77188b3020d6fe069fbc712f21aeec"
Nov 03 20:39:59.567710 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.567690    5591 scope.go:122] "RemoveContainer" containerID="c14f54124d5b880b893e9a8f2e0cea84cd77188b3020d6fe069fbc712f21aeec"
Nov 03 20:39:59.568007 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:39:59.567983    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c14f54124d5b880b893e9a8f2e0cea84cd77188b3020d6fe069fbc712f21aeec\": container with ID starting with c14f54124d5b880b893e9a8f2e0cea84cd77188b3020d6fe069fbc712f21aeec not found: ID does not exist" containerID="c14f54124d5b880b893e9a8f2e0cea84cd77188b3020d6fe069fbc712f21aeec"
Nov 03 20:39:59.568078 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.568017    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c14f54124d5b880b893e9a8f2e0cea84cd77188b3020d6fe069fbc712f21aeec"} err="failed to get container status \"c14f54124d5b880b893e9a8f2e0cea84cd77188b3020d6fe069fbc712f21aeec\": rpc error: code = NotFound desc = could not find container \"c14f54124d5b880b893e9a8f2e0cea84cd77188b3020d6fe069fbc712f21aeec\": container with ID starting with c14f54124d5b880b893e9a8f2e0cea84cd77188b3020d6fe069fbc712f21aeec not found: ID does not exist"
Nov 03 20:39:59.573045 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.573017    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v"]
Nov 03 20:39:59.579398 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.579372    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bfc240-5649698867-hsr2v"]
Nov 03 20:39:59.588973 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.588950    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp"]
Nov 03 20:39:59.593503 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.593481    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-bfc240-76778fd956-68nbp"]
Nov 03 20:39:59.913143 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.913035    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"]
Nov 03 20:39:59.913815 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:39:59.913638    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5" podUID="90331769-8442-4145-978d-63f3e1820556" containerName="istio-proxy" containerID="cri-o://605da35f2c95889c5931dd414bcfc14a985d180434b35d35a2c32f919c44d781" gracePeriod=30
Nov 03 20:40:00.151819 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:00.151782    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66"]
Nov 03 20:40:00.152011 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:00.151989    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66" podUID="ddf2f97d-ab36-416b-9c5d-29330b70c4b4" containerName="maas-api" containerID="cri-o://646f5ac5a3ee21e60f965643a80ab6c763625996bf71b998503618de3746b524" gracePeriod=30
Nov 03 20:40:00.469725 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:00.469694    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66"
Nov 03 20:40:00.525192 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:00.525153    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-maas-api-tls\") pod \"ddf2f97d-ab36-416b-9c5d-29330b70c4b4\" (UID: \"ddf2f97d-ab36-416b-9c5d-29330b70c4b4\") "
Nov 03 20:40:00.525391 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:00.525314    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-openshift-service-ca\") pod \"ddf2f97d-ab36-416b-9c5d-29330b70c4b4\" (UID: \"ddf2f97d-ab36-416b-9c5d-29330b70c4b4\") "
Nov 03 20:40:00.525467 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:00.525398    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-kube-api-access-vxz4t\" (UniqueName: \"kubernetes.io/projected/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-kube-api-access-vxz4t\") pod \"ddf2f97d-ab36-416b-9c5d-29330b70c4b4\" (UID: \"ddf2f97d-ab36-416b-9c5d-29330b70c4b4\") "
Nov 03 20:40:00.525539 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:00.525468    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-metrics-certs\") pod \"ddf2f97d-ab36-416b-9c5d-29330b70c4b4\" (UID: \"ddf2f97d-ab36-416b-9c5d-29330b70c4b4\") "
Nov 03 20:40:00.525688 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:00.525661    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-openshift-service-ca" pod "ddf2f97d-ab36-416b-9c5d-29330b70c4b4" (UID: "ddf2f97d-ab36-416b-9c5d-29330b70c4b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:40:00.526436 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:00.526408    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:00.527729 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:00.527667    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-maas-api-tls" pod "ddf2f97d-ab36-416b-9c5d-29330b70c4b4" (UID: "ddf2f97d-ab36-416b-9c5d-29330b70c4b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:40:00.527851 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:00.527776    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-kube-api-access-vxz4t" pod "ddf2f97d-ab36-416b-9c5d-29330b70c4b4" (UID: "ddf2f97d-ab36-416b-9c5d-29330b70c4b4"). InnerVolumeSpecName "kube-api-access-vxz4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:00.528283 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:00.528266    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-metrics-certs" pod "ddf2f97d-ab36-416b-9c5d-29330b70c4b4" (UID: "ddf2f97d-ab36-416b-9c5d-29330b70c4b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:40:00.557358 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:00.557326    5591 generic.go:356] "Generic (PLEG): container finished" podID="ddf2f97d-ab36-416b-9c5d-29330b70c4b4" containerID="646f5ac5a3ee21e60f965643a80ab6c763625996bf71b998503618de3746b524" exitCode=0
Nov 03 20:40:00.557506 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:00.557379    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66"
Nov 03 20:40:00.557506 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:00.557409    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66" event={"ID":"ddf2f97d-ab36-416b-9c5d-29330b70c4b4","Type":"ContainerDied","Data":"646f5ac5a3ee21e60f965643a80ab6c763625996bf71b998503618de3746b524"}
Nov 03 20:40:00.557506 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:00.557461    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66" event={"ID":"ddf2f97d-ab36-416b-9c5d-29330b70c4b4","Type":"ContainerDied","Data":"7d80d7948f37ad06fce7a6356a73ba0f20cfed8a2e107651813fcfecb66979c2"}
Nov 03 20:40:00.557506 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:00.557483    5591 scope.go:122] "RemoveContainer" containerID="646f5ac5a3ee21e60f965643a80ab6c763625996bf71b998503618de3746b524"
Nov 03 20:40:00.569095 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:00.569079    5591 scope.go:122] "RemoveContainer" containerID="646f5ac5a3ee21e60f965643a80ab6c763625996bf71b998503618de3746b524"
Nov 03 20:40:00.569383 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:40:00.569361    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"646f5ac5a3ee21e60f965643a80ab6c763625996bf71b998503618de3746b524\": container with ID starting with 646f5ac5a3ee21e60f965643a80ab6c763625996bf71b998503618de3746b524 not found: ID does not exist" containerID="646f5ac5a3ee21e60f965643a80ab6c763625996bf71b998503618de3746b524"
Nov 03 20:40:00.569478 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:00.569394    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"646f5ac5a3ee21e60f965643a80ab6c763625996bf71b998503618de3746b524"} err="failed to get container status \"646f5ac5a3ee21e60f965643a80ab6c763625996bf71b998503618de3746b524\": rpc error: code = NotFound desc = could not find container \"646f5ac5a3ee21e60f965643a80ab6c763625996bf71b998503618de3746b524\": container with ID starting with 646f5ac5a3ee21e60f965643a80ab6c763625996bf71b998503618de3746b524 not found: ID does not exist"
Nov 03 20:40:00.583391 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:00.583365    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66"]
Nov 03 20:40:00.590289 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:00.590264    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2b5703-5d67555749-jwk66"]
Nov 03 20:40:00.631270 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:00.630362    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:00.631270 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:00.630401    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxz4t\" (UniqueName: \"kubernetes.io/projected/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-kube-api-access-vxz4t\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:00.631270 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:00.630423    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddf2f97d-ab36-416b-9c5d-29330b70c4b4-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:01.350792 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:01.350755    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26612e45-5a14-497a-9e90-06f336b9d0d1" path="/var/lib/kubelet/pods/26612e45-5a14-497a-9e90-06f336b9d0d1/volumes"
Nov 03 20:40:01.351993 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:01.351966    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddf2f97d-ab36-416b-9c5d-29330b70c4b4" path="/var/lib/kubelet/pods/ddf2f97d-ab36-416b-9c5d-29330b70c4b4/volumes"
Nov 03 20:40:01.352706 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:01.352684    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3c18563-3261-4841-81d2-3db2ec91abd3" path="/var/lib/kubelet/pods/e3c18563-3261-4841-81d2-3db2ec91abd3/volumes"
Nov 03 20:40:02.011504 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:02.011466    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:40:03.488308 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.488273    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"]
Nov 03 20:40:03.489499 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.489467    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz" podUID="67d6b486-1932-4aec-ab87-770bb3f4db38" containerName="istio-proxy" containerID="cri-o://f93c7189b463f1c52bb4b99c31dbeb4edcc45522723255d8d4ce2833085a5d90" gracePeriod=30
Nov 03 20:40:03.709229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.709023    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"]
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.709821    5591 cpu_manager.go:414] "containerMap: removing container" podUID="7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455" containerName="payload-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.709839    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455" containerName="payload-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.709857    5591 cpu_manager.go:414] "containerMap: removing container" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.709866    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.709882    5591 cpu_manager.go:414] "containerMap: removing container" podUID="26612e45-5a14-497a-9e90-06f336b9d0d1" containerName="payload-pre-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.709891    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26612e45-5a14-497a-9e90-06f336b9d0d1" containerName="payload-pre-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.709907    5591 cpu_manager.go:414] "containerMap: removing container" podUID="d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb" containerName="maas-api"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.709915    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb" containerName="maas-api"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.709930    5591 cpu_manager.go:414] "containerMap: removing container" podUID="190ad217-4f23-4513-be1b-405c141d460d" containerName="payload-pre-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.709938    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="190ad217-4f23-4513-be1b-405c141d460d" containerName="payload-pre-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.709948    5591 cpu_manager.go:414] "containerMap: removing container" podUID="fe2c855d-4f89-4546-b580-e5f7e22eb9bc" containerName="payload-pre-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.709956    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe2c855d-4f89-4546-b580-e5f7e22eb9bc" containerName="payload-pre-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.709967    5591 cpu_manager.go:414] "containerMap: removing container" podUID="b7189d7d-2381-4085-94b8-5b6569b7da99" containerName="istio-proxy"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.709974    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7189d7d-2381-4085-94b8-5b6569b7da99" containerName="istio-proxy"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.709985    5591 cpu_manager.go:414] "containerMap: removing container" podUID="9d7c8f87-8009-4efa-a92f-3dd314f7bac0" containerName="maas-api"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.709991    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d7c8f87-8009-4efa-a92f-3dd314f7bac0" containerName="maas-api"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710003    5591 cpu_manager.go:414] "containerMap: removing container" podUID="080839df-c38a-4bfc-8b72-008a2c12de63" containerName="payload-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710010    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="080839df-c38a-4bfc-8b72-008a2c12de63" containerName="payload-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710021    5591 cpu_manager.go:414] "containerMap: removing container" podUID="b51ae7c4-0791-4250-809e-b558456a410b" containerName="payload-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710027    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b51ae7c4-0791-4250-809e-b558456a410b" containerName="payload-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710037    5591 cpu_manager.go:414] "containerMap: removing container" podUID="a67dbe17-4521-40e5-b4aa-602094676670" containerName="payload-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710044    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67dbe17-4521-40e5-b4aa-602094676670" containerName="payload-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710053    5591 cpu_manager.go:414] "containerMap: removing container" podUID="d26da545-16eb-415b-91ed-63db42e8f01d" containerName="payload-pre-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710060    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d26da545-16eb-415b-91ed-63db42e8f01d" containerName="payload-pre-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710074    5591 cpu_manager.go:414] "containerMap: removing container" podUID="7aff0e54-0f19-41b1-af95-70aee78fea74" containerName="istio-proxy"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710081    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aff0e54-0f19-41b1-af95-70aee78fea74" containerName="istio-proxy"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710090    5591 cpu_manager.go:414] "containerMap: removing container" podUID="83260a88-c456-4334-81ec-f640ab303cc3" containerName="istio-proxy"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710096    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83260a88-c456-4334-81ec-f640ab303cc3" containerName="istio-proxy"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710106    5591 cpu_manager.go:414] "containerMap: removing container" podUID="c9b4867c-02bd-4c72-8a46-6db1867a40c3" containerName="payload-pre-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710113    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9b4867c-02bd-4c72-8a46-6db1867a40c3" containerName="payload-pre-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710127    5591 cpu_manager.go:414] "containerMap: removing container" podUID="e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15" containerName="istio-proxy"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710134    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15" containerName="istio-proxy"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710145    5591 cpu_manager.go:414] "containerMap: removing container" podUID="d1031c5e-2ece-4213-ba23-33126946df5a" containerName="maas-api"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710153    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1031c5e-2ece-4213-ba23-33126946df5a" containerName="maas-api"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710172    5591 cpu_manager.go:414] "containerMap: removing container" podUID="fbb496cd-b12d-4331-af8b-417c6fbffdf2" containerName="maas-api"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710179    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb496cd-b12d-4331-af8b-417c6fbffdf2" containerName="maas-api"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710190    5591 cpu_manager.go:414] "containerMap: removing container" podUID="a884ccd6-a398-44d2-a3a7-40c8e96abd66" containerName="istio-proxy"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710196    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a884ccd6-a398-44d2-a3a7-40c8e96abd66" containerName="istio-proxy"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710220    5591 cpu_manager.go:414] "containerMap: removing container" podUID="57c85ce9-ebbf-4c18-b40c-e09ea4e9c068" containerName="main"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710228    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57c85ce9-ebbf-4c18-b40c-e09ea4e9c068" containerName="main"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710238    5591 cpu_manager.go:414] "containerMap: removing container" podUID="e3c18563-3261-4841-81d2-3db2ec91abd3" containerName="payload-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710244    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c18563-3261-4841-81d2-3db2ec91abd3" containerName="payload-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710253    5591 cpu_manager.go:414] "containerMap: removing container" podUID="df5c29d5-b63f-47ca-892a-d288dfa08f7f" containerName="maas-api"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710260    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df5c29d5-b63f-47ca-892a-d288dfa08f7f" containerName="maas-api"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710270    5591 cpu_manager.go:414] "containerMap: removing container" podUID="120358c4-5509-4cc4-8522-647ddbde2d90" containerName="istio-proxy"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710277    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="120358c4-5509-4cc4-8522-647ddbde2d90" containerName="istio-proxy"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710294    5591 cpu_manager.go:414] "containerMap: removing container" podUID="e10f31e1-5a95-4c6a-88ed-0d15291cb44e" containerName="main"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710300    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e10f31e1-5a95-4c6a-88ed-0d15291cb44e" containerName="main"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710315    5591 cpu_manager.go:414] "containerMap: removing container" podUID="824d78ad-b227-4bdd-8245-6f37ef395666" containerName="istio-proxy"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710322    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824d78ad-b227-4bdd-8245-6f37ef395666" containerName="istio-proxy"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710332    5591 cpu_manager.go:414] "containerMap: removing container" podUID="81aa9ef2-7d1b-488c-bbd2-fba606a67f5b" containerName="istio-proxy"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710339    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81aa9ef2-7d1b-488c-bbd2-fba606a67f5b" containerName="istio-proxy"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710350    5591 cpu_manager.go:414] "containerMap: removing container" podUID="9d9d77f5-8819-447e-b0a0-c27ce2d1d498" containerName="payload-pre-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710356    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d9d77f5-8819-447e-b0a0-c27ce2d1d498" containerName="payload-pre-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710365    5591 cpu_manager.go:414] "containerMap: removing container" podUID="22691b0c-82ca-4d24-85e6-fc861fcb73c0" containerName="maas-api"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710371    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22691b0c-82ca-4d24-85e6-fc861fcb73c0" containerName="maas-api"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710381    5591 cpu_manager.go:414] "containerMap: removing container" podUID="143283f7-8182-41d6-bff0-e171c14a2836" containerName="main"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710388    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="143283f7-8182-41d6-bff0-e171c14a2836" containerName="main"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710403    5591 cpu_manager.go:414] "containerMap: removing container" podUID="92c8f021-930c-438f-baf2-7457458d03af" containerName="maas-api"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710410    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92c8f021-930c-438f-baf2-7457458d03af" containerName="maas-api"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710426    5591 cpu_manager.go:414] "containerMap: removing container" podUID="ddf2f97d-ab36-416b-9c5d-29330b70c4b4" containerName="maas-api"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710432    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddf2f97d-ab36-416b-9c5d-29330b70c4b4" containerName="maas-api"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710453    5591 cpu_manager.go:414] "containerMap: removing container" podUID="86168259-d196-495e-9435-33d4f0e625d9" containerName="maas-api"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710461    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86168259-d196-495e-9435-33d4f0e625d9" containerName="maas-api"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710597    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb496cd-b12d-4331-af8b-417c6fbffdf2" containerName="maas-api"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710610    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="a884ccd6-a398-44d2-a3a7-40c8e96abd66" containerName="istio-proxy"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710621    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="57c85ce9-ebbf-4c18-b40c-e09ea4e9c068" containerName="main"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710633    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3c18563-3261-4841-81d2-3db2ec91abd3" containerName="payload-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710641    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="df5c29d5-b63f-47ca-892a-d288dfa08f7f" containerName="maas-api"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710648    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="120358c4-5509-4cc4-8522-647ddbde2d90" containerName="istio-proxy"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710659    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="e10f31e1-5a95-4c6a-88ed-0d15291cb44e" containerName="main"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710668    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="824d78ad-b227-4bdd-8245-6f37ef395666" containerName="istio-proxy"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710675    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="81aa9ef2-7d1b-488c-bbd2-fba606a67f5b" containerName="istio-proxy"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710685    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d9d77f5-8819-447e-b0a0-c27ce2d1d498" containerName="payload-pre-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710695    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="22691b0c-82ca-4d24-85e6-fc861fcb73c0" containerName="maas-api"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710704    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="143283f7-8182-41d6-bff0-e171c14a2836" containerName="main"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710714    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="92c8f021-930c-438f-baf2-7457458d03af" containerName="maas-api"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710722    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddf2f97d-ab36-416b-9c5d-29330b70c4b4" containerName="maas-api"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710731    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="86168259-d196-495e-9435-33d4f0e625d9" containerName="maas-api"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710739    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f3d9d49-2b92-4e5d-ba1c-f8e07ffd3455" containerName="payload-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710749    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="58f1605e-5db2-4a8d-9b62-4c84e113ac33" containerName="main"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710758    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="26612e45-5a14-497a-9e90-06f336b9d0d1" containerName="payload-pre-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710765    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0bf4e43-7c53-41e7-90be-4b3ca9d2dedb" containerName="maas-api"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710774    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="190ad217-4f23-4513-be1b-405c141d460d" containerName="payload-pre-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710782    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe2c855d-4f89-4546-b580-e5f7e22eb9bc" containerName="payload-pre-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710791    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7189d7d-2381-4085-94b8-5b6569b7da99" containerName="istio-proxy"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710799    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d7c8f87-8009-4efa-a92f-3dd314f7bac0" containerName="maas-api"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710807    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aff0e54-0f19-41b1-af95-70aee78fea74" containerName="istio-proxy"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710814    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="080839df-c38a-4bfc-8b72-008a2c12de63" containerName="payload-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710824    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="b51ae7c4-0791-4250-809e-b558456a410b" containerName="payload-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710831    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="a67dbe17-4521-40e5-b4aa-602094676670" containerName="payload-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710838    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="d26da545-16eb-415b-91ed-63db42e8f01d" containerName="payload-pre-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710845    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="83260a88-c456-4334-81ec-f640ab303cc3" containerName="istio-proxy"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710853    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9b4867c-02bd-4c72-8a46-6db1867a40c3" containerName="payload-pre-processing"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710862    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4a5d499-e4e8-4489-bbef-bc6e6ffd8b15" containerName="istio-proxy"
Nov 03 20:40:03.713223 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.710871    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1031c5e-2ece-4213-ba23-33126946df5a" containerName="maas-api"
Nov 03 20:40:03.762351 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.761419    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:03.762351 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.761318    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"]
Nov 03 20:40:03.891939 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.891894    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:03.892143 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.891972    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:03.892143 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.892128    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:03.892294 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.892162    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:03.892294 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.892191    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m57s\" (UniqueName: \"kubernetes.io/projected/328b01ad-1c34-473f-8e11-5e131775edc6-kube-api-access-6m57s\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:03.892294 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.892287    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/328b01ad-1c34-473f-8e11-5e131775edc6-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:03.892434 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.892323    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/328b01ad-1c34-473f-8e11-5e131775edc6-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:03.892483 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.892455    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:03.892527 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.892511    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/328b01ad-1c34-473f-8e11-5e131775edc6-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:03.993723 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.993694    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:03.993723 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.993725    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:03.993967 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.993803    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:03.993967 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.993818    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:03.993967 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.993838    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m57s\" (UniqueName: \"kubernetes.io/projected/328b01ad-1c34-473f-8e11-5e131775edc6-kube-api-access-6m57s\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:03.993967 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.993875    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/328b01ad-1c34-473f-8e11-5e131775edc6-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:03.994197 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.994026    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/328b01ad-1c34-473f-8e11-5e131775edc6-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:03.994197 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.994117    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:03.994197 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.994129    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-workload-certs\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:03.994197 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.994157    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/328b01ad-1c34-473f-8e11-5e131775edc6-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:03.994197 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.994189    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-credential-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:03.994479 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.994260    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-workload-socket\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:03.994479 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.994450    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-istio-data\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:03.994797 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.994773    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/328b01ad-1c34-473f-8e11-5e131775edc6-istiod-ca-cert\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:03.996364 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.996344    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-istio-envoy\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:03.996756 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:03.996737    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/328b01ad-1c34-473f-8e11-5e131775edc6-istio-podinfo\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:04.009384 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:04.009356    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/328b01ad-1c34-473f-8e11-5e131775edc6-istio-token\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:04.009497 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:04.009427    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m57s\" (UniqueName: \"kubernetes.io/projected/328b01ad-1c34-473f-8e11-5e131775edc6-kube-api-access-6m57s\") pod \"e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") " pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:04.083197 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:04.083163    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:04.405063 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:04.405036    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"]
Nov 03 20:40:04.406732 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:40:04.406705    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod328b01ad_1c34_473f_8e11_5e131775edc6.slice/crio-8db202938092fd91245260243a79e728d1f0682dd718453efaae0e3ed1e67637 WatchSource:0}: Error finding container 8db202938092fd91245260243a79e728d1f0682dd718453efaae0e3ed1e67637: Status 404 returned error can't find the container with id 8db202938092fd91245260243a79e728d1f0682dd718453efaae0e3ed1e67637
Nov 03 20:40:04.409123 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:04.409091    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:40:04.409247 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:04.409153    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:40:04.409247 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:04.409184    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:40:04.611772 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:04.611738    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks" event={"ID":"328b01ad-1c34-473f-8e11-5e131775edc6","Type":"ContainerStarted","Data":"314b84220b6acd5ed6e19f316cdcc073adf405b4a3ae04e4077b13a3662a53c8"}
Nov 03 20:40:04.612311 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:04.611791    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks" event={"ID":"328b01ad-1c34-473f-8e11-5e131775edc6","Type":"ContainerStarted","Data":"8db202938092fd91245260243a79e728d1f0682dd718453efaae0e3ed1e67637"}
Nov 03 20:40:04.636306 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:04.636233    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks" podStartSLOduration=1.636191875 podStartE2EDuration="1.636191875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:04.633627935 +0000 UTC m=+3753.405920957" watchObservedRunningTime="2026-11-03 20:40:04.636191875 +0000 UTC m=+3753.408484895"
Nov 03 20:40:04.714155 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:04.714068    5591 patch_prober.go:29] interesting pod/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:40:04.714155 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:04.714129    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz" podUID="67d6b486-1932-4aec-ab87-770bb3f4db38" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:40:05.083410 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.083369    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:05.085610 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.085583    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body=
Nov 03 20:40:05.085733 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.085654    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks" podUID="328b01ad-1c34-473f-8e11-5e131775edc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused"
Nov 03 20:40:05.208610 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.208575    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:40:05.313181 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.313146    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-istio-envoy\") pod \"90331769-8442-4145-978d-63f3e1820556\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") "
Nov 03 20:40:05.313397 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.313259    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-workload-socket\") pod \"90331769-8442-4145-978d-63f3e1820556\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") "
Nov 03 20:40:05.313397 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.313353    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90331769-8442-4145-978d-63f3e1820556-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90331769-8442-4145-978d-63f3e1820556-istiod-ca-cert\") pod \"90331769-8442-4145-978d-63f3e1820556\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") "
Nov 03 20:40:05.313397 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.313381    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90331769-8442-4145-978d-63f3e1820556-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90331769-8442-4145-978d-63f3e1820556-istio-podinfo\") pod \"90331769-8442-4145-978d-63f3e1820556\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") "
Nov 03 20:40:05.313602 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.313404    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-credential-socket\") pod \"90331769-8442-4145-978d-63f3e1820556\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") "
Nov 03 20:40:05.313602 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.313492    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-workload-certs\") pod \"90331769-8442-4145-978d-63f3e1820556\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") "
Nov 03 20:40:05.313602 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.313571    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-workload-socket" pod "90331769-8442-4145-978d-63f3e1820556" (UID: "90331769-8442-4145-978d-63f3e1820556"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:05.313763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.313595    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90331769-8442-4145-978d-63f3e1820556-kube-api-access-55d57\" (UniqueName: \"kubernetes.io/projected/90331769-8442-4145-978d-63f3e1820556-kube-api-access-55d57\") pod \"90331769-8442-4145-978d-63f3e1820556\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") "
Nov 03 20:40:05.313763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.313682    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-istio-data\") pod \"90331769-8442-4145-978d-63f3e1820556\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") "
Nov 03 20:40:05.313880 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.313772    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-credential-socket" pod "90331769-8442-4145-978d-63f3e1820556" (UID: "90331769-8442-4145-978d-63f3e1820556"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:05.313948 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.313876    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90331769-8442-4145-978d-63f3e1820556-istiod-ca-cert" pod "90331769-8442-4145-978d-63f3e1820556" (UID: "90331769-8442-4145-978d-63f3e1820556"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:40:05.313948 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.313929    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-istio-data" pod "90331769-8442-4145-978d-63f3e1820556" (UID: "90331769-8442-4145-978d-63f3e1820556"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:05.314063 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.313896    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-workload-certs" pod "90331769-8442-4145-978d-63f3e1820556" (UID: "90331769-8442-4145-978d-63f3e1820556"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:05.314125 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.313950    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90331769-8442-4145-978d-63f3e1820556-istio-token\" (UniqueName: \"kubernetes.io/projected/90331769-8442-4145-978d-63f3e1820556-istio-token\") pod \"90331769-8442-4145-978d-63f3e1820556\" (UID: \"90331769-8442-4145-978d-63f3e1820556\") "
Nov 03 20:40:05.314601 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.314579    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:05.314601 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.314601    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:05.314785 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.314610    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90331769-8442-4145-978d-63f3e1820556-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:05.314785 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.314619    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:05.314785 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.314628    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:05.315901 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.315870    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90331769-8442-4145-978d-63f3e1820556-istio-podinfo" pod "90331769-8442-4145-978d-63f3e1820556" (UID: "90331769-8442-4145-978d-63f3e1820556"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:40:05.315989 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.315956    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90331769-8442-4145-978d-63f3e1820556-kube-api-access-55d57" pod "90331769-8442-4145-978d-63f3e1820556" (UID: "90331769-8442-4145-978d-63f3e1820556"). InnerVolumeSpecName "kube-api-access-55d57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:05.316103 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.316083    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-istio-envoy" pod "90331769-8442-4145-978d-63f3e1820556" (UID: "90331769-8442-4145-978d-63f3e1820556"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:05.316330 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.316310    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90331769-8442-4145-978d-63f3e1820556-istio-token" pod "90331769-8442-4145-978d-63f3e1820556" (UID: "90331769-8442-4145-978d-63f3e1820556"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:05.341124 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.341093    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"]
Nov 03 20:40:05.381004 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.380970    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"]
Nov 03 20:40:05.416126 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.416096    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55d57\" (UniqueName: \"kubernetes.io/projected/90331769-8442-4145-978d-63f3e1820556-kube-api-access-55d57\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:05.416126 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.416122    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90331769-8442-4145-978d-63f3e1820556-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:05.416126 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.416133    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90331769-8442-4145-978d-63f3e1820556-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:05.416390 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.416142    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90331769-8442-4145-978d-63f3e1820556-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:05.620030 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.619943    5591 generic.go:356] "Generic (PLEG): container finished" podID="90331769-8442-4145-978d-63f3e1820556" containerID="605da35f2c95889c5931dd414bcfc14a985d180434b35d35a2c32f919c44d781" exitCode=0
Nov 03 20:40:05.620030 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.619998    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"
Nov 03 20:40:05.620631 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.620023    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5" event={"ID":"90331769-8442-4145-978d-63f3e1820556","Type":"ContainerDied","Data":"605da35f2c95889c5931dd414bcfc14a985d180434b35d35a2c32f919c44d781"}
Nov 03 20:40:05.620631 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.620081    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5" event={"ID":"90331769-8442-4145-978d-63f3e1820556","Type":"ContainerDied","Data":"ec1df8c1008343113b808700836aba7ed4f2954385ee805497e4a83c2a277332"}
Nov 03 20:40:05.620631 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.620095    5591 scope.go:122] "RemoveContainer" containerID="605da35f2c95889c5931dd414bcfc14a985d180434b35d35a2c32f919c44d781"
Nov 03 20:40:05.638051 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.638024    5591 scope.go:122] "RemoveContainer" containerID="605da35f2c95889c5931dd414bcfc14a985d180434b35d35a2c32f919c44d781"
Nov 03 20:40:05.638374 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:40:05.638353    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"605da35f2c95889c5931dd414bcfc14a985d180434b35d35a2c32f919c44d781\": container with ID starting with 605da35f2c95889c5931dd414bcfc14a985d180434b35d35a2c32f919c44d781 not found: ID does not exist" containerID="605da35f2c95889c5931dd414bcfc14a985d180434b35d35a2c32f919c44d781"
Nov 03 20:40:05.638442 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.638383    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"605da35f2c95889c5931dd414bcfc14a985d180434b35d35a2c32f919c44d781"} err="failed to get container status \"605da35f2c95889c5931dd414bcfc14a985d180434b35d35a2c32f919c44d781\": rpc error: code = NotFound desc = could not find container \"605da35f2c95889c5931dd414bcfc14a985d180434b35d35a2c32f919c44d781\": container with ID starting with 605da35f2c95889c5931dd414bcfc14a985d180434b35d35a2c32f919c44d781 not found: ID does not exist"
Nov 03 20:40:05.642813 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.642788    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"]
Nov 03 20:40:05.652016 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:05.651989    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-43c674-openshift-default-695d458dd9-rlzn5"]
Nov 03 20:40:06.084687 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:06.084647    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body=
Nov 03 20:40:06.085028 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:06.084998    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks" podUID="328b01ad-1c34-473f-8e11-5e131775edc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused"
Nov 03 20:40:07.092655 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:07.092614    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body=
Nov 03 20:40:07.093467 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:07.093434    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks" podUID="328b01ad-1c34-473f-8e11-5e131775edc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused"
Nov 03 20:40:07.349242 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:07.349155    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90331769-8442-4145-978d-63f3e1820556" path="/var/lib/kubelet/pods/90331769-8442-4145-978d-63f3e1820556/volumes"
Nov 03 20:40:08.089696 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.089650    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body=
Nov 03 20:40:08.090029 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.090000    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks" podUID="328b01ad-1c34-473f-8e11-5e131775edc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused"
Nov 03 20:40:08.643526 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.643493    5591 generic.go:356] "Generic (PLEG): container finished" podID="67d6b486-1932-4aec-ab87-770bb3f4db38" containerID="f93c7189b463f1c52bb4b99c31dbeb4edcc45522723255d8d4ce2833085a5d90" exitCode=0
Nov 03 20:40:08.643909 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.643575    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz" event={"ID":"67d6b486-1932-4aec-ab87-770bb3f4db38","Type":"ContainerDied","Data":"f93c7189b463f1c52bb4b99c31dbeb4edcc45522723255d8d4ce2833085a5d90"}
Nov 03 20:40:08.820667 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.820638    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:40:08.962599 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.962566    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-workload-socket\") pod \"67d6b486-1932-4aec-ab87-770bb3f4db38\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") "
Nov 03 20:40:08.963020 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.962873    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-workload-certs\") pod \"67d6b486-1932-4aec-ab87-770bb3f4db38\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") "
Nov 03 20:40:08.963020 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.962965    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67d6b486-1932-4aec-ab87-770bb3f4db38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d6b486-1932-4aec-ab87-770bb3f4db38-istiod-ca-cert\") pod \"67d6b486-1932-4aec-ab87-770bb3f4db38\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") "
Nov 03 20:40:08.963020 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.962989    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-data\") pod \"67d6b486-1932-4aec-ab87-770bb3f4db38\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") "
Nov 03 20:40:08.963320 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.963043    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-envoy\") pod \"67d6b486-1932-4aec-ab87-770bb3f4db38\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") "
Nov 03 20:40:08.963320 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.963077    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-credential-socket\") pod \"67d6b486-1932-4aec-ab87-770bb3f4db38\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") "
Nov 03 20:40:08.963320 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.963099    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-workload-socket" pod "67d6b486-1932-4aec-ab87-770bb3f4db38" (UID: "67d6b486-1932-4aec-ab87-770bb3f4db38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:08.963491 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.963319    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-podinfo\") pod \"67d6b486-1932-4aec-ab87-770bb3f4db38\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") "
Nov 03 20:40:08.963491 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.963372    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d6b486-1932-4aec-ab87-770bb3f4db38-kube-api-access-n9nv9\" (UniqueName: \"kubernetes.io/projected/67d6b486-1932-4aec-ab87-770bb3f4db38-kube-api-access-n9nv9\") pod \"67d6b486-1932-4aec-ab87-770bb3f4db38\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") "
Nov 03 20:40:08.963491 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.963413    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-token\" (UniqueName: \"kubernetes.io/projected/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-token\") pod \"67d6b486-1932-4aec-ab87-770bb3f4db38\" (UID: \"67d6b486-1932-4aec-ab87-770bb3f4db38\") "
Nov 03 20:40:08.963655 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.963533    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d6b486-1932-4aec-ab87-770bb3f4db38-istiod-ca-cert" pod "67d6b486-1932-4aec-ab87-770bb3f4db38" (UID: "67d6b486-1932-4aec-ab87-770bb3f4db38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:40:08.963787 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.963766    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-workload-certs" pod "67d6b486-1932-4aec-ab87-770bb3f4db38" (UID: "67d6b486-1932-4aec-ab87-770bb3f4db38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:08.963875 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.963814    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-credential-socket" pod "67d6b486-1932-4aec-ab87-770bb3f4db38" (UID: "67d6b486-1932-4aec-ab87-770bb3f4db38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:08.963962 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.963938    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-data" pod "67d6b486-1932-4aec-ab87-770bb3f4db38" (UID: "67d6b486-1932-4aec-ab87-770bb3f4db38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:08.964190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.964168    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:08.964328 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.964195    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:08.964328 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.964241    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67d6b486-1932-4aec-ab87-770bb3f4db38-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:08.964328 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.964254    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:08.964328 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.964268    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:08.966300 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.966275    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d6b486-1932-4aec-ab87-770bb3f4db38-kube-api-access-n9nv9" pod "67d6b486-1932-4aec-ab87-770bb3f4db38" (UID: "67d6b486-1932-4aec-ab87-770bb3f4db38"). InnerVolumeSpecName "kube-api-access-n9nv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:08.966407 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.966345    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-podinfo" pod "67d6b486-1932-4aec-ab87-770bb3f4db38" (UID: "67d6b486-1932-4aec-ab87-770bb3f4db38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:40:08.966407 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.966387    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-token" pod "67d6b486-1932-4aec-ab87-770bb3f4db38" (UID: "67d6b486-1932-4aec-ab87-770bb3f4db38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:08.966504 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:08.966391    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-envoy" pod "67d6b486-1932-4aec-ab87-770bb3f4db38" (UID: "67d6b486-1932-4aec-ab87-770bb3f4db38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:09.068628 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:09.067509    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:09.068628 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:09.067553    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:09.068628 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:09.067570    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9nv9\" (UniqueName: \"kubernetes.io/projected/67d6b486-1932-4aec-ab87-770bb3f4db38-kube-api-access-n9nv9\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:09.068628 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:09.067584    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67d6b486-1932-4aec-ab87-770bb3f4db38-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:09.089191 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:09.088718    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body=
Nov 03 20:40:09.089191 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:09.088791    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks" podUID="328b01ad-1c34-473f-8e11-5e131775edc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused"
Nov 03 20:40:09.651404 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:09.651317    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"
Nov 03 20:40:09.651885 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:09.651316    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz" event={"ID":"67d6b486-1932-4aec-ab87-770bb3f4db38","Type":"ContainerDied","Data":"dff926f37d46da6a44bf2a264aeb01d6075788cdf95fb7fefb5d3b712bf2187b"}
Nov 03 20:40:09.651885 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:09.651477    5591 scope.go:122] "RemoveContainer" containerID="f93c7189b463f1c52bb4b99c31dbeb4edcc45522723255d8d4ce2833085a5d90"
Nov 03 20:40:09.671862 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:09.671838    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"]
Nov 03 20:40:09.675925 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:09.675903    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bfc240-openshift-default-857dd6696c-s6mnz"]
Nov 03 20:40:09.834891 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:09.834859    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr"]
Nov 03 20:40:09.846247 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:09.846162    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc"]
Nov 03 20:40:10.097230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:10.085638    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body=
Nov 03 20:40:10.097230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:10.085709    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks" podUID="328b01ad-1c34-473f-8e11-5e131775edc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused"
Nov 03 20:40:10.470583 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:10.470500    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6"]
Nov 03 20:40:10.470819 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:10.470766    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6" podUID="762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2" containerName="maas-api" containerID="cri-o://f44858d5f24ac8656eb764d913bb3adfba01ce13dfaecc932fa4bbc9b1dddd49" gracePeriod=30
Nov 03 20:40:10.659571 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:10.659540    5591 generic.go:356] "Generic (PLEG): container finished" podID="762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2" containerID="f44858d5f24ac8656eb764d913bb3adfba01ce13dfaecc932fa4bbc9b1dddd49" exitCode=0
Nov 03 20:40:10.659964 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:10.659613    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6" event={"ID":"762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2","Type":"ContainerDied","Data":"f44858d5f24ac8656eb764d913bb3adfba01ce13dfaecc932fa4bbc9b1dddd49"}
Nov 03 20:40:10.936370 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:10.936343    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6"
Nov 03 20:40:11.018420 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:11.018389    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-maas-api-tls\") pod \"762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2\" (UID: \"762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2\") "
Nov 03 20:40:11.018644 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:11.018445    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-metrics-certs\") pod \"762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2\" (UID: \"762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2\") "
Nov 03 20:40:11.018644 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:11.018479    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-openshift-service-ca\") pod \"762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2\" (UID: \"762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2\") "
Nov 03 20:40:11.018644 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:11.018505    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-kube-api-access-hdzph\" (UniqueName: \"kubernetes.io/projected/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-kube-api-access-hdzph\") pod \"762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2\" (UID: \"762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2\") "
Nov 03 20:40:11.019011 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:11.018977    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-openshift-service-ca" pod "762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2" (UID: "762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:40:11.021584 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:11.021551    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-kube-api-access-hdzph" pod "762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2" (UID: "762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2"). InnerVolumeSpecName "kube-api-access-hdzph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:11.023525 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:11.023497    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-maas-api-tls" pod "762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2" (UID: "762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:40:11.023525 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:11.023516    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-metrics-certs" pod "762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2" (UID: "762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:40:11.103702 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:11.103660    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body=
Nov 03 20:40:11.104023 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:11.103978    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks" podUID="328b01ad-1c34-473f-8e11-5e131775edc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused"
Nov 03 20:40:11.119763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:11.119729    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:11.119763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:11.119759    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:11.119956 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:11.119773    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:11.119956 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:11.119783    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdzph\" (UniqueName: \"kubernetes.io/projected/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2-kube-api-access-hdzph\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:11.349758 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:11.349724    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67d6b486-1932-4aec-ab87-770bb3f4db38" path="/var/lib/kubelet/pods/67d6b486-1932-4aec-ab87-770bb3f4db38/volumes"
Nov 03 20:40:11.669216 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:11.669118    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6"
Nov 03 20:40:11.669216 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:11.669125    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6" event={"ID":"762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2","Type":"ContainerDied","Data":"fe38fb0572686e0f36d1f85ebdedba9e8ea873fa8c9175bf5b638b3197972a90"}
Nov 03 20:40:11.669793 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:11.669270    5591 scope.go:122] "RemoveContainer" containerID="f44858d5f24ac8656eb764d913bb3adfba01ce13dfaecc932fa4bbc9b1dddd49"
Nov 03 20:40:11.691221 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:11.691137    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6"]
Nov 03 20:40:11.693558 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:11.693531    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcfde8-6576968c6f-dddq6"]
Nov 03 20:40:12.083587 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:12.083541    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body=
Nov 03 20:40:12.083815 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:12.083600    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks" podUID="328b01ad-1c34-473f-8e11-5e131775edc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused"
Nov 03 20:40:13.088008 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:13.087970    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:13.088524 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:13.088499    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:13.089334 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:13.089313    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:13.349228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:13.349130    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2" path="/var/lib/kubelet/pods/762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2/volumes"
Nov 03 20:40:14.835324 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:14.835265    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr" podUID="122f235f-b27f-4342-8f25-f004502a0291" containerName="payload-processing" containerID="cri-o://0271911efb89677e8334b8fd2dbf53ed5491b9960c6af61e8149894e5c5a8d56" gracePeriod=25
Nov 03 20:40:14.846842 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:14.846800    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc" podUID="c85b7cb7-d242-4015-8bd9-bd6a583fae15" containerName="payload-pre-processing" containerID="cri-o://073b8b2fc768b07b20f218f3edb204a02c33a489bb41c7449071c989ebd991ae" gracePeriod=25
Nov 03 20:40:15.125297 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.125256    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr"
Nov 03 20:40:15.167186 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.167026    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc"
Nov 03 20:40:15.185269 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.185233    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/122f235f-b27f-4342-8f25-f004502a0291-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122f235f-b27f-4342-8f25-f004502a0291-plugins-config-volume\") pod \"122f235f-b27f-4342-8f25-f004502a0291\" (UID: \"122f235f-b27f-4342-8f25-f004502a0291\") "
Nov 03 20:40:15.185536 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.185514    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/122f235f-b27f-4342-8f25-f004502a0291-kube-api-access-jh6kr\" (UniqueName: \"kubernetes.io/projected/122f235f-b27f-4342-8f25-f004502a0291-kube-api-access-jh6kr\") pod \"122f235f-b27f-4342-8f25-f004502a0291\" (UID: \"122f235f-b27f-4342-8f25-f004502a0291\") "
Nov 03 20:40:15.186017 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.185920    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/122f235f-b27f-4342-8f25-f004502a0291-plugins-config-volume" pod "122f235f-b27f-4342-8f25-f004502a0291" (UID: "122f235f-b27f-4342-8f25-f004502a0291"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:40:15.186146 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.186055    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122f235f-b27f-4342-8f25-f004502a0291-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:15.188648 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.188619    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/122f235f-b27f-4342-8f25-f004502a0291-kube-api-access-jh6kr" pod "122f235f-b27f-4342-8f25-f004502a0291" (UID: "122f235f-b27f-4342-8f25-f004502a0291"). InnerVolumeSpecName "kube-api-access-jh6kr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:15.287260 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.287195    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c85b7cb7-d242-4015-8bd9-bd6a583fae15-kube-api-access-lsf7v\" (UniqueName: \"kubernetes.io/projected/c85b7cb7-d242-4015-8bd9-bd6a583fae15-kube-api-access-lsf7v\") pod \"c85b7cb7-d242-4015-8bd9-bd6a583fae15\" (UID: \"c85b7cb7-d242-4015-8bd9-bd6a583fae15\") "
Nov 03 20:40:15.287445 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.287376    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c85b7cb7-d242-4015-8bd9-bd6a583fae15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85b7cb7-d242-4015-8bd9-bd6a583fae15-plugins-config-volume\") pod \"c85b7cb7-d242-4015-8bd9-bd6a583fae15\" (UID: \"c85b7cb7-d242-4015-8bd9-bd6a583fae15\") "
Nov 03 20:40:15.287679 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.287655    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh6kr\" (UniqueName: \"kubernetes.io/projected/122f235f-b27f-4342-8f25-f004502a0291-kube-api-access-jh6kr\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:15.287856 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.287722    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c85b7cb7-d242-4015-8bd9-bd6a583fae15-plugins-config-volume" pod "c85b7cb7-d242-4015-8bd9-bd6a583fae15" (UID: "c85b7cb7-d242-4015-8bd9-bd6a583fae15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:40:15.289404 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.289384    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85b7cb7-d242-4015-8bd9-bd6a583fae15-kube-api-access-lsf7v" pod "c85b7cb7-d242-4015-8bd9-bd6a583fae15" (UID: "c85b7cb7-d242-4015-8bd9-bd6a583fae15"). InnerVolumeSpecName "kube-api-access-lsf7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:15.388777 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.388742    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85b7cb7-d242-4015-8bd9-bd6a583fae15-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:15.388777 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.388772    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsf7v\" (UniqueName: \"kubernetes.io/projected/c85b7cb7-d242-4015-8bd9-bd6a583fae15-kube-api-access-lsf7v\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:15.697434 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.697346    5591 generic.go:356] "Generic (PLEG): container finished" podID="122f235f-b27f-4342-8f25-f004502a0291" containerID="0271911efb89677e8334b8fd2dbf53ed5491b9960c6af61e8149894e5c5a8d56" exitCode=0
Nov 03 20:40:15.697434 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.697401    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr"
Nov 03 20:40:15.697671 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.697436    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr" event={"ID":"122f235f-b27f-4342-8f25-f004502a0291","Type":"ContainerDied","Data":"0271911efb89677e8334b8fd2dbf53ed5491b9960c6af61e8149894e5c5a8d56"}
Nov 03 20:40:15.697671 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.697490    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr" event={"ID":"122f235f-b27f-4342-8f25-f004502a0291","Type":"ContainerDied","Data":"d11ef053c5d94301ee9b6b7def451f0ce15af01d5e03c06d5b4fde65db512dbc"}
Nov 03 20:40:15.697671 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.697517    5591 scope.go:122] "RemoveContainer" containerID="0271911efb89677e8334b8fd2dbf53ed5491b9960c6af61e8149894e5c5a8d56"
Nov 03 20:40:15.699045 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.699009    5591 generic.go:356] "Generic (PLEG): container finished" podID="c85b7cb7-d242-4015-8bd9-bd6a583fae15" containerID="073b8b2fc768b07b20f218f3edb204a02c33a489bb41c7449071c989ebd991ae" exitCode=0
Nov 03 20:40:15.699352 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.699052    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc"
Nov 03 20:40:15.699352 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.699059    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc" event={"ID":"c85b7cb7-d242-4015-8bd9-bd6a583fae15","Type":"ContainerDied","Data":"073b8b2fc768b07b20f218f3edb204a02c33a489bb41c7449071c989ebd991ae"}
Nov 03 20:40:15.699352 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.699102    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc" event={"ID":"c85b7cb7-d242-4015-8bd9-bd6a583fae15","Type":"ContainerDied","Data":"ea450e875dab50413e7ad3749c02565b306a9b5093dab2d1df5ed5a86a5334e0"}
Nov 03 20:40:15.708165 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.708145    5591 scope.go:122] "RemoveContainer" containerID="0271911efb89677e8334b8fd2dbf53ed5491b9960c6af61e8149894e5c5a8d56"
Nov 03 20:40:15.708494 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:40:15.708475    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0271911efb89677e8334b8fd2dbf53ed5491b9960c6af61e8149894e5c5a8d56\": container with ID starting with 0271911efb89677e8334b8fd2dbf53ed5491b9960c6af61e8149894e5c5a8d56 not found: ID does not exist" containerID="0271911efb89677e8334b8fd2dbf53ed5491b9960c6af61e8149894e5c5a8d56"
Nov 03 20:40:15.708594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.708502    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0271911efb89677e8334b8fd2dbf53ed5491b9960c6af61e8149894e5c5a8d56"} err="failed to get container status \"0271911efb89677e8334b8fd2dbf53ed5491b9960c6af61e8149894e5c5a8d56\": rpc error: code = NotFound desc = could not find container \"0271911efb89677e8334b8fd2dbf53ed5491b9960c6af61e8149894e5c5a8d56\": container with ID starting with 0271911efb89677e8334b8fd2dbf53ed5491b9960c6af61e8149894e5c5a8d56 not found: ID does not exist"
Nov 03 20:40:15.708594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.708518    5591 scope.go:122] "RemoveContainer" containerID="073b8b2fc768b07b20f218f3edb204a02c33a489bb41c7449071c989ebd991ae"
Nov 03 20:40:15.718449 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.718430    5591 scope.go:122] "RemoveContainer" containerID="073b8b2fc768b07b20f218f3edb204a02c33a489bb41c7449071c989ebd991ae"
Nov 03 20:40:15.718748 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:40:15.718726    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"073b8b2fc768b07b20f218f3edb204a02c33a489bb41c7449071c989ebd991ae\": container with ID starting with 073b8b2fc768b07b20f218f3edb204a02c33a489bb41c7449071c989ebd991ae not found: ID does not exist" containerID="073b8b2fc768b07b20f218f3edb204a02c33a489bb41c7449071c989ebd991ae"
Nov 03 20:40:15.718861 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.718756    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"073b8b2fc768b07b20f218f3edb204a02c33a489bb41c7449071c989ebd991ae"} err="failed to get container status \"073b8b2fc768b07b20f218f3edb204a02c33a489bb41c7449071c989ebd991ae\": rpc error: code = NotFound desc = could not find container \"073b8b2fc768b07b20f218f3edb204a02c33a489bb41c7449071c989ebd991ae\": container with ID starting with 073b8b2fc768b07b20f218f3edb204a02c33a489bb41c7449071c989ebd991ae not found: ID does not exist"
Nov 03 20:40:15.720529 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.720504    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc"]
Nov 03 20:40:15.723082 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.723056    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:40:15.725790 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.725770    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-bcfde8-78bcbccdf9-pbsmc"]
Nov 03 20:40:15.739929 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.739896    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr"]
Nov 03 20:40:15.745523 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:15.745499    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bcfde8-95f87c9cb-bxvrr"]
Nov 03 20:40:17.351845 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:17.351811    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="122f235f-b27f-4342-8f25-f004502a0291" path="/var/lib/kubelet/pods/122f235f-b27f-4342-8f25-f004502a0291/volumes"
Nov 03 20:40:17.352326 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:17.352307    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c85b7cb7-d242-4015-8bd9-bd6a583fae15" path="/var/lib/kubelet/pods/c85b7cb7-d242-4015-8bd9-bd6a583fae15/volumes"
Nov 03 20:40:20.364752 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.364690    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" containerID="cri-o://cf92c48529a2152532f83b4375a75e7a963ef5d1cd7ed9e0ce9ac5bc8b75ec76" gracePeriod=45
Nov 03 20:40:20.402437 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.402399    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq" podUID="8c68cd56-eb04-4f8e-806a-510cc9ec2644" containerName="main" containerID="cri-o://567ed7d5563fc18a5b8f47314ca0abb1c13bb535e34e7f81c908c7be1adecf02" gracePeriod=45
Nov 03 20:40:20.500818 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.500773    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": dial tcp 10.134.0.74:8000: connect: connection refused"
Nov 03 20:40:20.510246 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.505961    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"]
Nov 03 20:40:20.510246 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.506555    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk" podUID="2e20e21d-9530-4cdb-9927-1376cc371094" containerName="istio-proxy" containerID="cri-o://011d89e379e254ef23e9388d9bd2cd6ed574910cad0153ae7562280c867affb8" gracePeriod=30
Nov 03 20:40:20.761633 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.760640    5591 generic.go:356] "Generic (PLEG): container finished" podID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerID="cf92c48529a2152532f83b4375a75e7a963ef5d1cd7ed9e0ce9ac5bc8b75ec76" exitCode=0
Nov 03 20:40:20.761633 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.760802    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" event={"ID":"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0","Type":"ContainerDied","Data":"cf92c48529a2152532f83b4375a75e7a963ef5d1cd7ed9e0ce9ac5bc8b75ec76"}
Nov 03 20:40:20.761633 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.760845    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2" event={"ID":"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0","Type":"ContainerDied","Data":"495b2d71efb3a69b016f26696396a72e849667a9c6a1e4444c17e7f7184af2a3"}
Nov 03 20:40:20.761633 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.760862    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="495b2d71efb3a69b016f26696396a72e849667a9c6a1e4444c17e7f7184af2a3"
Nov 03 20:40:20.763440 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.763409    5591 generic.go:356] "Generic (PLEG): container finished" podID="8c68cd56-eb04-4f8e-806a-510cc9ec2644" containerID="567ed7d5563fc18a5b8f47314ca0abb1c13bb535e34e7f81c908c7be1adecf02" exitCode=0
Nov 03 20:40:20.763567 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.763480    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq" event={"ID":"8c68cd56-eb04-4f8e-806a-510cc9ec2644","Type":"ContainerDied","Data":"567ed7d5563fc18a5b8f47314ca0abb1c13bb535e34e7f81c908c7be1adecf02"}
Nov 03 20:40:20.764377 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.764352    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 20:40:20.802351 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.802323    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"
Nov 03 20:40:20.892611 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.892581    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-kube-api-access-jpwsw\" (UniqueName: \"kubernetes.io/projected/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-kube-api-access-jpwsw\") pod \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\" (UID: \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\") "
Nov 03 20:40:20.892835 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.892629    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-model-cache\") pod \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\" (UID: \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\") "
Nov 03 20:40:20.892835 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.892777    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-tls-certs\" (UniqueName: \"kubernetes.io/secret/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-tls-certs\") pod \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\" (UID: \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\") "
Nov 03 20:40:20.892835 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.892810    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-tmp-dir\") pod \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\" (UID: \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\") "
Nov 03 20:40:20.893011 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.892943    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-home\" (UniqueName: \"kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-home\") pod \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\" (UID: \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\") "
Nov 03 20:40:20.893011 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.892955    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-model-cache" pod "a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" (UID: "a240d836-cfa9-48a2-9bb0-ad49ed9da4d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:20.893011 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.892986    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-dshm\") pod \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\" (UID: \"a240d836-cfa9-48a2-9bb0-ad49ed9da4d0\") "
Nov 03 20:40:20.893580 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.893555    5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-model-cache\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:20.894636 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.894608    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-home" pod "a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" (UID: "a240d836-cfa9-48a2-9bb0-ad49ed9da4d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:20.895081 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.895050    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-tmp-dir" pod "a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" (UID: "a240d836-cfa9-48a2-9bb0-ad49ed9da4d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:20.895688 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.895662    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-kube-api-access-jpwsw" pod "a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" (UID: "a240d836-cfa9-48a2-9bb0-ad49ed9da4d0"). InnerVolumeSpecName "kube-api-access-jpwsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:20.895899 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.895873    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-tls-certs" pod "a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" (UID: "a240d836-cfa9-48a2-9bb0-ad49ed9da4d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:40:20.895983 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.895964    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-dshm" pod "a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" (UID: "a240d836-cfa9-48a2-9bb0-ad49ed9da4d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:20.994761 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.994681    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c68cd56-eb04-4f8e-806a-510cc9ec2644-tls-certs\" (UniqueName: \"kubernetes.io/secret/8c68cd56-eb04-4f8e-806a-510cc9ec2644-tls-certs\") pod \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\" (UID: \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\") "
Nov 03 20:40:20.994912 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.994785    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-model-cache\") pod \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\" (UID: \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\") "
Nov 03 20:40:20.994912 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.994815    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-home\" (UniqueName: \"kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-home\") pod \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\" (UID: \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\") "
Nov 03 20:40:20.994912 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.994868    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-dshm\") pod \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\" (UID: \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\") "
Nov 03 20:40:20.994912 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.994902    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-tmp-dir\") pod \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\" (UID: \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\") "
Nov 03 20:40:20.995099 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.994939    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c68cd56-eb04-4f8e-806a-510cc9ec2644-kube-api-access-6kcws\" (UniqueName: \"kubernetes.io/projected/8c68cd56-eb04-4f8e-806a-510cc9ec2644-kube-api-access-6kcws\") pod \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\" (UID: \"8c68cd56-eb04-4f8e-806a-510cc9ec2644\") "
Nov 03 20:40:20.995099 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.995034    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-model-cache" pod "8c68cd56-eb04-4f8e-806a-510cc9ec2644" (UID: "8c68cd56-eb04-4f8e-806a-510cc9ec2644"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:20.995099 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.995045    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-home" pod "8c68cd56-eb04-4f8e-806a-510cc9ec2644" (UID: "8c68cd56-eb04-4f8e-806a-510cc9ec2644"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:20.995303 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.995113    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-tmp-dir" pod "8c68cd56-eb04-4f8e-806a-510cc9ec2644" (UID: "8c68cd56-eb04-4f8e-806a-510cc9ec2644"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:20.995492 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.995465    5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-tls-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:20.995492 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.995491    5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-tmp-dir\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:20.995492 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.995499    5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-home\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:20.995904 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.995510    5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-dshm\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:20.995904 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.995524    5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-model-cache\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:20.995904 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.995534    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpwsw\" (UniqueName: \"kubernetes.io/projected/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0-kube-api-access-jpwsw\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:20.995904 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.995542    5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-home\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:20.995904 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.995550    5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-tmp-dir\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:20.996995 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.996970    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c68cd56-eb04-4f8e-806a-510cc9ec2644-tls-certs" pod "8c68cd56-eb04-4f8e-806a-510cc9ec2644" (UID: "8c68cd56-eb04-4f8e-806a-510cc9ec2644"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:40:20.997265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.997184    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-dshm" pod "8c68cd56-eb04-4f8e-806a-510cc9ec2644" (UID: "8c68cd56-eb04-4f8e-806a-510cc9ec2644"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:20.997265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:20.997196    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c68cd56-eb04-4f8e-806a-510cc9ec2644-kube-api-access-6kcws" pod "8c68cd56-eb04-4f8e-806a-510cc9ec2644" (UID: "8c68cd56-eb04-4f8e-806a-510cc9ec2644"). InnerVolumeSpecName "kube-api-access-6kcws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:21.096801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:21.096767    5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c68cd56-eb04-4f8e-806a-510cc9ec2644-dshm\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:21.096801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:21.096798    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kcws\" (UniqueName: \"kubernetes.io/projected/8c68cd56-eb04-4f8e-806a-510cc9ec2644-kube-api-access-6kcws\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:21.096801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:21.096809    5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c68cd56-eb04-4f8e-806a-510cc9ec2644-tls-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:21.721030 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:21.720919    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq" podUID="8c68cd56-eb04-4f8e-806a-510cc9ec2644" containerName="main" probeResult="failure" output="Get \"https://10.134.1.11:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:40:21.771617 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:21.771581    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"
Nov 03 20:40:21.771617 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:21.771600    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq" event={"ID":"8c68cd56-eb04-4f8e-806a-510cc9ec2644","Type":"ContainerDied","Data":"54122ecc923c054938b621c6d40090c2393dd899036c0a0c39878bb72ec141fa"}
Nov 03 20:40:21.771919 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:21.771664    5591 scope.go:122] "RemoveContainer" containerID="567ed7d5563fc18a5b8f47314ca0abb1c13bb535e34e7f81c908c7be1adecf02"
Nov 03 20:40:21.771919 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:21.771841    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"
Nov 03 20:40:21.802245 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:21.801636    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"]
Nov 03 20:40:21.804253 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:21.804228    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/distinct-2-w1-32e482-kserve-5ffbf94589-b4mg2"]
Nov 03 20:40:21.825964 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:21.825934    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"]
Nov 03 20:40:21.830042 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:21.830015    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-32e482/simulator-w1-32e482-kserve-776bf6595d-qlczq"]
Nov 03 20:40:23.348800 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:23.348767    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c68cd56-eb04-4f8e-806a-510cc9ec2644" path="/var/lib/kubelet/pods/8c68cd56-eb04-4f8e-806a-510cc9ec2644/volumes"
Nov 03 20:40:23.349234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:23.349173    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" path="/var/lib/kubelet/pods/a240d836-cfa9-48a2-9bb0-ad49ed9da4d0/volumes"
Nov 03 20:40:23.725180 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:23.725088    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"]
Nov 03 20:40:23.725381 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:23.725354    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr" podUID="916d26e6-67e4-49d7-9df8-18c7087aaac1" containerName="discovery" containerID="cri-o://ac9a2aaa57f42c88a64a76c82f813b610a1dee7deddbff3cdae84db95c6794f9" gracePeriod=30
Nov 03 20:40:24.014079 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.014055    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"
Nov 03 20:40:24.031968 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.031935    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-csr-dns-cert\") pod \"916d26e6-67e4-49d7-9df8-18c7087aaac1\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") "
Nov 03 20:40:24.032130 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.032093    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/916d26e6-67e4-49d7-9df8-18c7087aaac1-cacerts\" (UniqueName: \"kubernetes.io/secret/916d26e6-67e4-49d7-9df8-18c7087aaac1-cacerts\") pod \"916d26e6-67e4-49d7-9df8-18c7087aaac1\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") "
Nov 03 20:40:24.032174 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.032158    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-kubeconfig\") pod \"916d26e6-67e4-49d7-9df8-18c7087aaac1\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") "
Nov 03 20:40:24.032267 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.032187    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/916d26e6-67e4-49d7-9df8-18c7087aaac1-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/916d26e6-67e4-49d7-9df8-18c7087aaac1-local-certs\") pod \"916d26e6-67e4-49d7-9df8-18c7087aaac1\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") "
Nov 03 20:40:24.032267 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.032238    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-token\" (UniqueName: \"kubernetes.io/projected/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-token\") pod \"916d26e6-67e4-49d7-9df8-18c7087aaac1\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") "
Nov 03 20:40:24.032385 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.032345    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-csr-ca-configmap\") pod \"916d26e6-67e4-49d7-9df8-18c7087aaac1\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") "
Nov 03 20:40:24.032447 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.032409    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/916d26e6-67e4-49d7-9df8-18c7087aaac1-kube-api-access-kfrvn\" (UniqueName: \"kubernetes.io/projected/916d26e6-67e4-49d7-9df8-18c7087aaac1-kube-api-access-kfrvn\") pod \"916d26e6-67e4-49d7-9df8-18c7087aaac1\" (UID: \"916d26e6-67e4-49d7-9df8-18c7087aaac1\") "
Nov 03 20:40:24.033910 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.033875    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-csr-ca-configmap" pod "916d26e6-67e4-49d7-9df8-18c7087aaac1" (UID: "916d26e6-67e4-49d7-9df8-18c7087aaac1"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:40:24.036185 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.034974    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-csr-dns-cert" pod "916d26e6-67e4-49d7-9df8-18c7087aaac1" (UID: "916d26e6-67e4-49d7-9df8-18c7087aaac1"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:40:24.036185 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.035100    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/916d26e6-67e4-49d7-9df8-18c7087aaac1-local-certs" pod "916d26e6-67e4-49d7-9df8-18c7087aaac1" (UID: "916d26e6-67e4-49d7-9df8-18c7087aaac1"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:24.036185 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.035486    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-token" pod "916d26e6-67e4-49d7-9df8-18c7087aaac1" (UID: "916d26e6-67e4-49d7-9df8-18c7087aaac1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:24.036185 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.035583    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-kubeconfig" pod "916d26e6-67e4-49d7-9df8-18c7087aaac1" (UID: "916d26e6-67e4-49d7-9df8-18c7087aaac1"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:40:24.036185 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.035628    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/916d26e6-67e4-49d7-9df8-18c7087aaac1-cacerts" pod "916d26e6-67e4-49d7-9df8-18c7087aaac1" (UID: "916d26e6-67e4-49d7-9df8-18c7087aaac1"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:40:24.036570 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.036553    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916d26e6-67e4-49d7-9df8-18c7087aaac1-kube-api-access-kfrvn" pod "916d26e6-67e4-49d7-9df8-18c7087aaac1" (UID: "916d26e6-67e4-49d7-9df8-18c7087aaac1"). InnerVolumeSpecName "kube-api-access-kfrvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:24.134355 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.134311    5591 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-csr-dns-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:24.134355 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.134350    5591 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/916d26e6-67e4-49d7-9df8-18c7087aaac1-cacerts\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:24.134595 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.134364    5591 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-kubeconfig\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:24.134595 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.134379    5591 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/916d26e6-67e4-49d7-9df8-18c7087aaac1-local-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:24.134595 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.134391    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:24.134595 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.134405    5591 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/916d26e6-67e4-49d7-9df8-18c7087aaac1-istio-csr-ca-configmap\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:24.134595 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.134417    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfrvn\" (UniqueName: \"kubernetes.io/projected/916d26e6-67e4-49d7-9df8-18c7087aaac1-kube-api-access-kfrvn\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:24.794115 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.794081    5591 generic.go:356] "Generic (PLEG): container finished" podID="916d26e6-67e4-49d7-9df8-18c7087aaac1" containerID="ac9a2aaa57f42c88a64a76c82f813b610a1dee7deddbff3cdae84db95c6794f9" exitCode=0
Nov 03 20:40:24.794581 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.794129    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"
Nov 03 20:40:24.794581 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.794174    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr" event={"ID":"916d26e6-67e4-49d7-9df8-18c7087aaac1","Type":"ContainerDied","Data":"ac9a2aaa57f42c88a64a76c82f813b610a1dee7deddbff3cdae84db95c6794f9"}
Nov 03 20:40:24.794581 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.794260    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr" event={"ID":"916d26e6-67e4-49d7-9df8-18c7087aaac1","Type":"ContainerDied","Data":"83a0cfa24a5287bb6b017044b716a35860dcfa55b706c42385861ffd4ca7c6b3"}
Nov 03 20:40:24.794581 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.794287    5591 scope.go:122] "RemoveContainer" containerID="ac9a2aaa57f42c88a64a76c82f813b610a1dee7deddbff3cdae84db95c6794f9"
Nov 03 20:40:24.805608 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.805591    5591 scope.go:122] "RemoveContainer" containerID="ac9a2aaa57f42c88a64a76c82f813b610a1dee7deddbff3cdae84db95c6794f9"
Nov 03 20:40:24.805888 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:40:24.805867    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac9a2aaa57f42c88a64a76c82f813b610a1dee7deddbff3cdae84db95c6794f9\": container with ID starting with ac9a2aaa57f42c88a64a76c82f813b610a1dee7deddbff3cdae84db95c6794f9 not found: ID does not exist" containerID="ac9a2aaa57f42c88a64a76c82f813b610a1dee7deddbff3cdae84db95c6794f9"
Nov 03 20:40:24.805944 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.805898    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac9a2aaa57f42c88a64a76c82f813b610a1dee7deddbff3cdae84db95c6794f9"} err="failed to get container status \"ac9a2aaa57f42c88a64a76c82f813b610a1dee7deddbff3cdae84db95c6794f9\": rpc error: code = NotFound desc = could not find container \"ac9a2aaa57f42c88a64a76c82f813b610a1dee7deddbff3cdae84db95c6794f9\": container with ID starting with ac9a2aaa57f42c88a64a76c82f813b610a1dee7deddbff3cdae84db95c6794f9 not found: ID does not exist"
Nov 03 20:40:24.817750 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.817727    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"]
Nov 03 20:40:24.822883 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:24.822861    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lcqfr"]
Nov 03 20:40:25.349910 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.349871    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="916d26e6-67e4-49d7-9df8-18c7087aaac1" path="/var/lib/kubelet/pods/916d26e6-67e4-49d7-9df8-18c7087aaac1/volumes"
Nov 03 20:40:25.387340 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.387303    5591 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:40:25.387524 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.387374    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk" podUID="2e20e21d-9530-4cdb-9927-1376cc371094" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:40:25.774237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.774184    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:40:25.807141 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.807107    5591 generic.go:356] "Generic (PLEG): container finished" podID="2e20e21d-9530-4cdb-9927-1376cc371094" containerID="011d89e379e254ef23e9388d9bd2cd6ed574910cad0153ae7562280c867affb8" exitCode=0
Nov 03 20:40:25.807673 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.807176    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"
Nov 03 20:40:25.807673 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.807186    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk" event={"ID":"2e20e21d-9530-4cdb-9927-1376cc371094","Type":"ContainerDied","Data":"011d89e379e254ef23e9388d9bd2cd6ed574910cad0153ae7562280c867affb8"}
Nov 03 20:40:25.807673 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.807282    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk" event={"ID":"2e20e21d-9530-4cdb-9927-1376cc371094","Type":"ContainerDied","Data":"bd895216e5ceb77f3175cd63edaf928db942b99c6edf5ee373e3a4ebcb9762be"}
Nov 03 20:40:25.807673 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.807299    5591 scope.go:122] "RemoveContainer" containerID="011d89e379e254ef23e9388d9bd2cd6ed574910cad0153ae7562280c867affb8"
Nov 03 20:40:25.819864 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.819843    5591 scope.go:122] "RemoveContainer" containerID="011d89e379e254ef23e9388d9bd2cd6ed574910cad0153ae7562280c867affb8"
Nov 03 20:40:25.820161 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:40:25.820137    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"011d89e379e254ef23e9388d9bd2cd6ed574910cad0153ae7562280c867affb8\": container with ID starting with 011d89e379e254ef23e9388d9bd2cd6ed574910cad0153ae7562280c867affb8 not found: ID does not exist" containerID="011d89e379e254ef23e9388d9bd2cd6ed574910cad0153ae7562280c867affb8"
Nov 03 20:40:25.820271 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.820175    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"011d89e379e254ef23e9388d9bd2cd6ed574910cad0153ae7562280c867affb8"} err="failed to get container status \"011d89e379e254ef23e9388d9bd2cd6ed574910cad0153ae7562280c867affb8\": rpc error: code = NotFound desc = could not find container \"011d89e379e254ef23e9388d9bd2cd6ed574910cad0153ae7562280c867affb8\": container with ID starting with 011d89e379e254ef23e9388d9bd2cd6ed574910cad0153ae7562280c867affb8 not found: ID does not exist"
Nov 03 20:40:25.853863 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.853839    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-credential-socket\") pod \"2e20e21d-9530-4cdb-9927-1376cc371094\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") "
Nov 03 20:40:25.854026 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.853895    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e20e21d-9530-4cdb-9927-1376cc371094-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e20e21d-9530-4cdb-9927-1376cc371094-istiod-ca-cert\") pod \"2e20e21d-9530-4cdb-9927-1376cc371094\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") "
Nov 03 20:40:25.854026 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.853920    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-istio-envoy\") pod \"2e20e21d-9530-4cdb-9927-1376cc371094\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") "
Nov 03 20:40:25.854026 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.853947    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-workload-certs\") pod \"2e20e21d-9530-4cdb-9927-1376cc371094\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") "
Nov 03 20:40:25.854026 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.853981    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e20e21d-9530-4cdb-9927-1376cc371094-istio-token\" (UniqueName: \"kubernetes.io/projected/2e20e21d-9530-4cdb-9927-1376cc371094-istio-token\") pod \"2e20e21d-9530-4cdb-9927-1376cc371094\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") "
Nov 03 20:40:25.854277 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.854066    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-istio-data\") pod \"2e20e21d-9530-4cdb-9927-1376cc371094\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") "
Nov 03 20:40:25.854277 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.854099    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e20e21d-9530-4cdb-9927-1376cc371094-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e20e21d-9530-4cdb-9927-1376cc371094-istio-podinfo\") pod \"2e20e21d-9530-4cdb-9927-1376cc371094\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") "
Nov 03 20:40:25.854277 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.854107    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-credential-socket" pod "2e20e21d-9530-4cdb-9927-1376cc371094" (UID: "2e20e21d-9530-4cdb-9927-1376cc371094"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:25.854277 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.854154    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e20e21d-9530-4cdb-9927-1376cc371094-kube-api-access-qfptp\" (UniqueName: \"kubernetes.io/projected/2e20e21d-9530-4cdb-9927-1376cc371094-kube-api-access-qfptp\") pod \"2e20e21d-9530-4cdb-9927-1376cc371094\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") "
Nov 03 20:40:25.854277 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.854231    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-workload-socket\") pod \"2e20e21d-9530-4cdb-9927-1376cc371094\" (UID: \"2e20e21d-9530-4cdb-9927-1376cc371094\") "
Nov 03 20:40:25.854564 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.854278    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-workload-certs" pod "2e20e21d-9530-4cdb-9927-1376cc371094" (UID: "2e20e21d-9530-4cdb-9927-1376cc371094"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:25.854564 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.854342    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-istio-data" pod "2e20e21d-9530-4cdb-9927-1376cc371094" (UID: "2e20e21d-9530-4cdb-9927-1376cc371094"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:25.854564 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.854374    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e20e21d-9530-4cdb-9927-1376cc371094-istiod-ca-cert" pod "2e20e21d-9530-4cdb-9927-1376cc371094" (UID: "2e20e21d-9530-4cdb-9927-1376cc371094"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:40:25.854752 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.854591    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-workload-socket" pod "2e20e21d-9530-4cdb-9927-1376cc371094" (UID: "2e20e21d-9530-4cdb-9927-1376cc371094"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:25.855076 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.854997    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:25.855076 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.855021    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e20e21d-9530-4cdb-9927-1376cc371094-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:25.855076 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.855040    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:25.855076 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.855054    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:25.855076 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.855068    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:25.856666 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.856632    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e20e21d-9530-4cdb-9927-1376cc371094-istio-podinfo" pod "2e20e21d-9530-4cdb-9927-1376cc371094" (UID: "2e20e21d-9530-4cdb-9927-1376cc371094"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:40:25.856666 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.856648    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e20e21d-9530-4cdb-9927-1376cc371094-istio-token" pod "2e20e21d-9530-4cdb-9927-1376cc371094" (UID: "2e20e21d-9530-4cdb-9927-1376cc371094"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:25.856890 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.856696    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-istio-envoy" pod "2e20e21d-9530-4cdb-9927-1376cc371094" (UID: "2e20e21d-9530-4cdb-9927-1376cc371094"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:25.856890 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.856730    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e20e21d-9530-4cdb-9927-1376cc371094-kube-api-access-qfptp" pod "2e20e21d-9530-4cdb-9927-1376cc371094" (UID: "2e20e21d-9530-4cdb-9927-1376cc371094"). InnerVolumeSpecName "kube-api-access-qfptp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:25.956497 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.956413    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e20e21d-9530-4cdb-9927-1376cc371094-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:25.956497 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.956454    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e20e21d-9530-4cdb-9927-1376cc371094-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:25.956497 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.956470    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e20e21d-9530-4cdb-9927-1376cc371094-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:25.956497 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.956485    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfptp\" (UniqueName: \"kubernetes.io/projected/2e20e21d-9530-4cdb-9927-1376cc371094-kube-api-access-qfptp\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:25.964267 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.964170    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"]
Nov 03 20:40:25.976821 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.976787    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"]
Nov 03 20:40:25.979792 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:25.979751    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"]
Nov 03 20:40:26.131513 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:26.131480    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"]
Nov 03 20:40:26.135946 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:26.135920    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bcfde8-openshift-default-69fd758c55-kz6hk"]
Nov 03 20:40:27.013340 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:27.013305    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:40:27.352633 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:27.352597    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e20e21d-9530-4cdb-9927-1376cc371094" path="/var/lib/kubelet/pods/2e20e21d-9530-4cdb-9927-1376cc371094/volumes"
Nov 03 20:40:27.695284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:27.694450    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv"]
Nov 03 20:40:27.697335 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:27.694761    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv" podUID="7e07d216-4ac1-414a-9803-a20d8a8bc2c8" containerName="maas-api" containerID="cri-o://32e2539d9c51ecf98736ecadf4c59aeb8caa03ee8c196f1d861d74c62bcd8050" gracePeriod=30
Nov 03 20:40:27.897234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:27.896542    5591 generic.go:356] "Generic (PLEG): container finished" podID="7e07d216-4ac1-414a-9803-a20d8a8bc2c8" containerID="32e2539d9c51ecf98736ecadf4c59aeb8caa03ee8c196f1d861d74c62bcd8050" exitCode=0
Nov 03 20:40:27.897234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:27.896704    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv" event={"ID":"7e07d216-4ac1-414a-9803-a20d8a8bc2c8","Type":"ContainerDied","Data":"32e2539d9c51ecf98736ecadf4c59aeb8caa03ee8c196f1d861d74c62bcd8050"}
Nov 03 20:40:28.037402 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:28.037372    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv"
Nov 03 20:40:28.090677 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:28.090492    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-metrics-certs\") pod \"7e07d216-4ac1-414a-9803-a20d8a8bc2c8\" (UID: \"7e07d216-4ac1-414a-9803-a20d8a8bc2c8\") "
Nov 03 20:40:28.090677 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:28.090635    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-openshift-service-ca\") pod \"7e07d216-4ac1-414a-9803-a20d8a8bc2c8\" (UID: \"7e07d216-4ac1-414a-9803-a20d8a8bc2c8\") "
Nov 03 20:40:28.090967 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:28.090755    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-kube-api-access-vhd4j\" (UniqueName: \"kubernetes.io/projected/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-kube-api-access-vhd4j\") pod \"7e07d216-4ac1-414a-9803-a20d8a8bc2c8\" (UID: \"7e07d216-4ac1-414a-9803-a20d8a8bc2c8\") "
Nov 03 20:40:28.090967 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:28.090794    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-maas-api-tls\") pod \"7e07d216-4ac1-414a-9803-a20d8a8bc2c8\" (UID: \"7e07d216-4ac1-414a-9803-a20d8a8bc2c8\") "
Nov 03 20:40:28.092100 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:28.092066    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-openshift-service-ca" pod "7e07d216-4ac1-414a-9803-a20d8a8bc2c8" (UID: "7e07d216-4ac1-414a-9803-a20d8a8bc2c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:40:28.094051 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:28.093887    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-metrics-certs" pod "7e07d216-4ac1-414a-9803-a20d8a8bc2c8" (UID: "7e07d216-4ac1-414a-9803-a20d8a8bc2c8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:40:28.094051 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:28.094003    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-maas-api-tls" pod "7e07d216-4ac1-414a-9803-a20d8a8bc2c8" (UID: "7e07d216-4ac1-414a-9803-a20d8a8bc2c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:40:28.094293 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:28.094265    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-kube-api-access-vhd4j" pod "7e07d216-4ac1-414a-9803-a20d8a8bc2c8" (UID: "7e07d216-4ac1-414a-9803-a20d8a8bc2c8"). InnerVolumeSpecName "kube-api-access-vhd4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:28.192586 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:28.192551    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhd4j\" (UniqueName: \"kubernetes.io/projected/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-kube-api-access-vhd4j\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:28.192586 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:28.192583    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:28.192586 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:28.192595    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:28.192586 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:28.192604    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e07d216-4ac1-414a-9803-a20d8a8bc2c8-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:28.749761 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:40:28.749721    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:40:28.911626 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:28.911581    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv" event={"ID":"7e07d216-4ac1-414a-9803-a20d8a8bc2c8","Type":"ContainerDied","Data":"7ed0723c6d73a728327ececfa6e53914504361ebb86b23a8f9b3e26d4999060b"}
Nov 03 20:40:28.911846 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:28.911666    5591 scope.go:122] "RemoveContainer" containerID="32e2539d9c51ecf98736ecadf4c59aeb8caa03ee8c196f1d861d74c62bcd8050"
Nov 03 20:40:28.913475 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:28.913357    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv"
Nov 03 20:40:28.951760 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:28.951483    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv"]
Nov 03 20:40:28.953650 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:28.953627    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-32e482-59cdc4f7d5-9nvsv"]
Nov 03 20:40:29.348372 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:29.348339    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e07d216-4ac1-414a-9803-a20d8a8bc2c8" path="/var/lib/kubelet/pods/7e07d216-4ac1-414a-9803-a20d8a8bc2c8/volumes"
Nov 03 20:40:37.104307 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:37.104267    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:40:37.798272 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:37.794049    5591 scope.go:122] "RemoveContainer" containerID="cf92c48529a2152532f83b4375a75e7a963ef5d1cd7ed9e0ce9ac5bc8b75ec76"
Nov 03 20:40:37.815275 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:37.815248    5591 scope.go:122] "RemoveContainer" containerID="72aa9d5a68cb141b51757eed21f00436a942681092448f149bf2bdec38a2e8f7"
Nov 03 20:40:40.997990 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:40.997929    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main" containerID="cri-o://0353ac9841442ed0b2b459195ab45ca668367f62a25e9870a28e1a8e48004e72" gracePeriod=45
Nov 03 20:40:41.003556 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.003504    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main" containerID="cri-o://e3b01a953a5f0a40565252edadff3c23cb9b38a7481930778869d0e03761c66d" gracePeriod=45
Nov 03 20:40:41.007461 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.007432    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh" podUID="665623ee-f9a4-4aae-922b-f229460a9cf2" containerName="main" containerID="cri-o://036a29cf2b55fe80c230491d6447b63f0f5cabb2fe4ebffa140e25fad7587edf" gracePeriod=45
Nov 03 20:40:41.280820 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.280257    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 20:40:41.303001 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.302938    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 20:40:41.311924 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.307606    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"
Nov 03 20:40:41.467874 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.467844    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/665623ee-f9a4-4aae-922b-f229460a9cf2-tls-certs\" (UniqueName: \"kubernetes.io/secret/665623ee-f9a4-4aae-922b-f229460a9cf2-tls-certs\") pod \"665623ee-f9a4-4aae-922b-f229460a9cf2\" (UID: \"665623ee-f9a4-4aae-922b-f229460a9cf2\") "
Nov 03 20:40:41.468080 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.467888    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-dshm\") pod \"665623ee-f9a4-4aae-922b-f229460a9cf2\" (UID: \"665623ee-f9a4-4aae-922b-f229460a9cf2\") "
Nov 03 20:40:41.468080 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.467911    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-home\" (UniqueName: \"kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-home\") pod \"9d797881-05fe-4105-9dbe-e7ab7b768982\" (UID: \"9d797881-05fe-4105-9dbe-e7ab7b768982\") "
Nov 03 20:40:41.468080 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.467931    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-dshm\") pod \"9d797881-05fe-4105-9dbe-e7ab7b768982\" (UID: \"9d797881-05fe-4105-9dbe-e7ab7b768982\") "
Nov 03 20:40:41.468080 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.467953    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-model-cache\") pod \"8cf0c23f-8862-442a-a487-35f4023789b8\" (UID: \"8cf0c23f-8862-442a-a487-35f4023789b8\") "
Nov 03 20:40:41.468080 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.467992    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cf0c23f-8862-442a-a487-35f4023789b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/8cf0c23f-8862-442a-a487-35f4023789b8-tls-certs\") pod \"8cf0c23f-8862-442a-a487-35f4023789b8\" (UID: \"8cf0c23f-8862-442a-a487-35f4023789b8\") "
Nov 03 20:40:41.468080 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.468030    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-tmp-dir\") pod \"9d797881-05fe-4105-9dbe-e7ab7b768982\" (UID: \"9d797881-05fe-4105-9dbe-e7ab7b768982\") "
Nov 03 20:40:41.468080 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.468071    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-model-cache\") pod \"9d797881-05fe-4105-9dbe-e7ab7b768982\" (UID: \"9d797881-05fe-4105-9dbe-e7ab7b768982\") "
Nov 03 20:40:41.468555 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.468113    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d797881-05fe-4105-9dbe-e7ab7b768982-kube-api-access-4t4zd\" (UniqueName: \"kubernetes.io/projected/9d797881-05fe-4105-9dbe-e7ab7b768982-kube-api-access-4t4zd\") pod \"9d797881-05fe-4105-9dbe-e7ab7b768982\" (UID: \"9d797881-05fe-4105-9dbe-e7ab7b768982\") "
Nov 03 20:40:41.468555 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.468158    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-home\" (UniqueName: \"kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-home\") pod \"665623ee-f9a4-4aae-922b-f229460a9cf2\" (UID: \"665623ee-f9a4-4aae-922b-f229460a9cf2\") "
Nov 03 20:40:41.468555 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.468234    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cf0c23f-8862-442a-a487-35f4023789b8-kube-api-access-vz79h\" (UniqueName: \"kubernetes.io/projected/8cf0c23f-8862-442a-a487-35f4023789b8-kube-api-access-vz79h\") pod \"8cf0c23f-8862-442a-a487-35f4023789b8\" (UID: \"8cf0c23f-8862-442a-a487-35f4023789b8\") "
Nov 03 20:40:41.468555 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.468276    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-model-cache\") pod \"665623ee-f9a4-4aae-922b-f229460a9cf2\" (UID: \"665623ee-f9a4-4aae-922b-f229460a9cf2\") "
Nov 03 20:40:41.468555 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.468309    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-home" pod "9d797881-05fe-4105-9dbe-e7ab7b768982" (UID: "9d797881-05fe-4105-9dbe-e7ab7b768982"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:41.468555 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.468328    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-tmp-dir\") pod \"665623ee-f9a4-4aae-922b-f229460a9cf2\" (UID: \"665623ee-f9a4-4aae-922b-f229460a9cf2\") "
Nov 03 20:40:41.468555 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.468333    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-model-cache" pod "8cf0c23f-8862-442a-a487-35f4023789b8" (UID: "8cf0c23f-8862-442a-a487-35f4023789b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:41.468555 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.468368    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/665623ee-f9a4-4aae-922b-f229460a9cf2-kube-api-access-vqw6n\" (UniqueName: \"kubernetes.io/projected/665623ee-f9a4-4aae-922b-f229460a9cf2-kube-api-access-vqw6n\") pod \"665623ee-f9a4-4aae-922b-f229460a9cf2\" (UID: \"665623ee-f9a4-4aae-922b-f229460a9cf2\") "
Nov 03 20:40:41.468555 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.468401    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-home\" (UniqueName: \"kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-home\") pod \"8cf0c23f-8862-442a-a487-35f4023789b8\" (UID: \"8cf0c23f-8862-442a-a487-35f4023789b8\") "
Nov 03 20:40:41.468555 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.468452    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-dshm\") pod \"8cf0c23f-8862-442a-a487-35f4023789b8\" (UID: \"8cf0c23f-8862-442a-a487-35f4023789b8\") "
Nov 03 20:40:41.468555 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.468520    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-tmp-dir\") pod \"8cf0c23f-8862-442a-a487-35f4023789b8\" (UID: \"8cf0c23f-8862-442a-a487-35f4023789b8\") "
Nov 03 20:40:41.468555 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.468520    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-tmp-dir" pod "9d797881-05fe-4105-9dbe-e7ab7b768982" (UID: "9d797881-05fe-4105-9dbe-e7ab7b768982"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:41.469475 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.468565    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d797881-05fe-4105-9dbe-e7ab7b768982-tls-certs\" (UniqueName: \"kubernetes.io/secret/9d797881-05fe-4105-9dbe-e7ab7b768982-tls-certs\") pod \"9d797881-05fe-4105-9dbe-e7ab7b768982\" (UID: \"9d797881-05fe-4105-9dbe-e7ab7b768982\") "
Nov 03 20:40:41.469475 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.469097    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-home" pod "665623ee-f9a4-4aae-922b-f229460a9cf2" (UID: "665623ee-f9a4-4aae-922b-f229460a9cf2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:41.469475 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.469253    5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-home\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:41.469475 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.469263    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-tmp-dir" pod "665623ee-f9a4-4aae-922b-f229460a9cf2" (UID: "665623ee-f9a4-4aae-922b-f229460a9cf2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:41.469475 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.469274    5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-model-cache\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:41.469475 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.469314    5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-tmp-dir\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:41.469475 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.469331    5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-home\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:41.469475 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.469446    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-model-cache" pod "9d797881-05fe-4105-9dbe-e7ab7b768982" (UID: "9d797881-05fe-4105-9dbe-e7ab7b768982"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:41.469475 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.469455    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-home" pod "8cf0c23f-8862-442a-a487-35f4023789b8" (UID: "8cf0c23f-8862-442a-a487-35f4023789b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:41.469992 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.469622    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-tmp-dir" pod "8cf0c23f-8862-442a-a487-35f4023789b8" (UID: "8cf0c23f-8862-442a-a487-35f4023789b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:41.469992 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.469735    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-model-cache" pod "665623ee-f9a4-4aae-922b-f229460a9cf2" (UID: "665623ee-f9a4-4aae-922b-f229460a9cf2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:41.471016 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.470981    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-dshm" pod "9d797881-05fe-4105-9dbe-e7ab7b768982" (UID: "9d797881-05fe-4105-9dbe-e7ab7b768982"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:41.471176 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.471025    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-dshm" pod "665623ee-f9a4-4aae-922b-f229460a9cf2" (UID: "665623ee-f9a4-4aae-922b-f229460a9cf2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:41.471911 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.471882    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/665623ee-f9a4-4aae-922b-f229460a9cf2-tls-certs" pod "665623ee-f9a4-4aae-922b-f229460a9cf2" (UID: "665623ee-f9a4-4aae-922b-f229460a9cf2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:40:41.472173 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.472145    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf0c23f-8862-442a-a487-35f4023789b8-kube-api-access-vz79h" pod "8cf0c23f-8862-442a-a487-35f4023789b8" (UID: "8cf0c23f-8862-442a-a487-35f4023789b8"). InnerVolumeSpecName "kube-api-access-vz79h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:41.472173 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.472158    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-dshm" pod "8cf0c23f-8862-442a-a487-35f4023789b8" (UID: "8cf0c23f-8862-442a-a487-35f4023789b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:41.472324 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.472251    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d797881-05fe-4105-9dbe-e7ab7b768982-kube-api-access-4t4zd" pod "9d797881-05fe-4105-9dbe-e7ab7b768982" (UID: "9d797881-05fe-4105-9dbe-e7ab7b768982"). InnerVolumeSpecName "kube-api-access-4t4zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:41.472324 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.472277    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/665623ee-f9a4-4aae-922b-f229460a9cf2-kube-api-access-vqw6n" pod "665623ee-f9a4-4aae-922b-f229460a9cf2" (UID: "665623ee-f9a4-4aae-922b-f229460a9cf2"). InnerVolumeSpecName "kube-api-access-vqw6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:41.472614 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.472600    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cf0c23f-8862-442a-a487-35f4023789b8-tls-certs" pod "8cf0c23f-8862-442a-a487-35f4023789b8" (UID: "8cf0c23f-8862-442a-a487-35f4023789b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:40:41.472967 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.472952    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d797881-05fe-4105-9dbe-e7ab7b768982-tls-certs" pod "9d797881-05fe-4105-9dbe-e7ab7b768982" (UID: "9d797881-05fe-4105-9dbe-e7ab7b768982"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:40:41.558389 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.558350    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4"]
Nov 03 20:40:41.558841 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.558827    5591 cpu_manager.go:414] "containerMap: removing container" podUID="2e20e21d-9530-4cdb-9927-1376cc371094" containerName="istio-proxy"
Nov 03 20:40:41.558893 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.558842    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e20e21d-9530-4cdb-9927-1376cc371094" containerName="istio-proxy"
Nov 03 20:40:41.558893 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.558853    5591 cpu_manager.go:414] "containerMap: removing container" podUID="c85b7cb7-d242-4015-8bd9-bd6a583fae15" containerName="payload-pre-processing"
Nov 03 20:40:41.558893 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.558858    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c85b7cb7-d242-4015-8bd9-bd6a583fae15" containerName="payload-pre-processing"
Nov 03 20:40:41.558893 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.558870    5591 cpu_manager.go:414] "containerMap: removing container" podUID="7e07d216-4ac1-414a-9803-a20d8a8bc2c8" containerName="maas-api"
Nov 03 20:40:41.558893 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.558875    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e07d216-4ac1-414a-9803-a20d8a8bc2c8" containerName="maas-api"
Nov 03 20:40:41.558893 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.558883    5591 cpu_manager.go:414] "containerMap: removing container" podUID="122f235f-b27f-4342-8f25-f004502a0291" containerName="payload-processing"
Nov 03 20:40:41.558893 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.558887    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="122f235f-b27f-4342-8f25-f004502a0291" containerName="payload-processing"
Nov 03 20:40:41.558893 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.558892    5591 cpu_manager.go:414] "containerMap: removing container" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.558897    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.558906    5591 cpu_manager.go:414] "containerMap: removing container" podUID="8c68cd56-eb04-4f8e-806a-510cc9ec2644" containerName="main"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.558910    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c68cd56-eb04-4f8e-806a-510cc9ec2644" containerName="main"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.558922    5591 cpu_manager.go:414] "containerMap: removing container" podUID="916d26e6-67e4-49d7-9df8-18c7087aaac1" containerName="discovery"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.558927    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="916d26e6-67e4-49d7-9df8-18c7087aaac1" containerName="discovery"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.558941    5591 cpu_manager.go:414] "containerMap: removing container" podUID="90331769-8442-4145-978d-63f3e1820556" containerName="istio-proxy"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.558945    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90331769-8442-4145-978d-63f3e1820556" containerName="istio-proxy"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.558953    5591 cpu_manager.go:414] "containerMap: removing container" podUID="67d6b486-1932-4aec-ab87-770bb3f4db38" containerName="istio-proxy"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.558958    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d6b486-1932-4aec-ab87-770bb3f4db38" containerName="istio-proxy"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.558971    5591 cpu_manager.go:414] "containerMap: removing container" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.558975    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.558983    5591 cpu_manager.go:414] "containerMap: removing container" podUID="762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2" containerName="maas-api"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.558987    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2" containerName="maas-api"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.558998    5591 cpu_manager.go:414] "containerMap: removing container" podUID="665623ee-f9a4-4aae-922b-f229460a9cf2" containerName="main"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.559002    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="665623ee-f9a4-4aae-922b-f229460a9cf2" containerName="main"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.559011    5591 cpu_manager.go:414] "containerMap: removing container" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.559015    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.559072    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="122f235f-b27f-4342-8f25-f004502a0291" containerName="payload-processing"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.559081    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerName="main"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.559087    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c68cd56-eb04-4f8e-806a-510cc9ec2644" containerName="main"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.559093    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="916d26e6-67e4-49d7-9df8-18c7087aaac1" containerName="discovery"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.559100    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="90331769-8442-4145-978d-63f3e1820556" containerName="istio-proxy"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.559106    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d6b486-1932-4aec-ab87-770bb3f4db38" containerName="istio-proxy"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.559114    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" containerName="main"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.559119    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="762e5146-9a8e-49d6-b2a8-d3c9e46cfcd2" containerName="maas-api"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.559125    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="665623ee-f9a4-4aae-922b-f229460a9cf2" containerName="main"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.559130    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="a240d836-cfa9-48a2-9bb0-ad49ed9da4d0" containerName="main"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.559135    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e20e21d-9530-4cdb-9927-1376cc371094" containerName="istio-proxy"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.559141    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c85b7cb7-d242-4015-8bd9-bd6a583fae15" containerName="payload-pre-processing"
Nov 03 20:40:41.559190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.559145    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e07d216-4ac1-414a-9803-a20d8a8bc2c8" containerName="maas-api"
Nov 03 20:40:41.563895 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.563874    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4"
Nov 03 20:40:41.569557 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.569262    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:40:41.569557 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.569404    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\""
Nov 03 20:40:41.570763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.570560    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t4zd\" (UniqueName: \"kubernetes.io/projected/9d797881-05fe-4105-9dbe-e7ab7b768982-kube-api-access-4t4zd\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:41.570763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.570584    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz79h\" (UniqueName: \"kubernetes.io/projected/8cf0c23f-8862-442a-a487-35f4023789b8-kube-api-access-vz79h\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:41.570763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.570600    5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-model-cache\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:41.570763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.570614    5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-tmp-dir\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:41.570763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.570628    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqw6n\" (UniqueName: \"kubernetes.io/projected/665623ee-f9a4-4aae-922b-f229460a9cf2-kube-api-access-vqw6n\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:41.570763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.570642    5591 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-home\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:41.570763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.570654    5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-dshm\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:41.570763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.570669    5591 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cf0c23f-8862-442a-a487-35f4023789b8-tmp-dir\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:41.570763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.570682    5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d797881-05fe-4105-9dbe-e7ab7b768982-tls-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:41.570763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.570694    5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/665623ee-f9a4-4aae-922b-f229460a9cf2-tls-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:41.570763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.570706    5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/665623ee-f9a4-4aae-922b-f229460a9cf2-dshm\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:41.570763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.570720    5591 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-dshm\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:41.570763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.570734    5591 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cf0c23f-8862-442a-a487-35f4023789b8-tls-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:41.570763 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.570746    5591 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d797881-05fe-4105-9dbe-e7ab7b768982-model-cache\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:41.571780 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.570861    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4"]
Nov 03 20:40:41.672482 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.672229    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7880a3e1-aceb-4444-9615-bf168854a75b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4\" (UID: \"7880a3e1-aceb-4444-9615-bf168854a75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4"
Nov 03 20:40:41.672482 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.672279    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5dx9\" (UniqueName: \"kubernetes.io/projected/7880a3e1-aceb-4444-9615-bf168854a75b-kube-api-access-h5dx9\") pod \"maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4\" (UID: \"7880a3e1-aceb-4444-9615-bf168854a75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4"
Nov 03 20:40:41.672821 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.672797    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7880a3e1-aceb-4444-9615-bf168854a75b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4\" (UID: \"7880a3e1-aceb-4444-9615-bf168854a75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4"
Nov 03 20:40:41.774284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.774247    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7880a3e1-aceb-4444-9615-bf168854a75b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4\" (UID: \"7880a3e1-aceb-4444-9615-bf168854a75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4"
Nov 03 20:40:41.774477 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.774420    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7880a3e1-aceb-4444-9615-bf168854a75b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4\" (UID: \"7880a3e1-aceb-4444-9615-bf168854a75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4"
Nov 03 20:40:41.774477 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.774448    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5dx9\" (UniqueName: \"kubernetes.io/projected/7880a3e1-aceb-4444-9615-bf168854a75b-kube-api-access-h5dx9\") pod \"maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4\" (UID: \"7880a3e1-aceb-4444-9615-bf168854a75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4"
Nov 03 20:40:41.775071 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.775044    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7880a3e1-aceb-4444-9615-bf168854a75b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4\" (UID: \"7880a3e1-aceb-4444-9615-bf168854a75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4"
Nov 03 20:40:41.776920 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.776900    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7880a3e1-aceb-4444-9615-bf168854a75b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4\" (UID: \"7880a3e1-aceb-4444-9615-bf168854a75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4"
Nov 03 20:40:41.797231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.797187    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5dx9\" (UniqueName: \"kubernetes.io/projected/7880a3e1-aceb-4444-9615-bf168854a75b-kube-api-access-h5dx9\") pod \"maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4\" (UID: \"7880a3e1-aceb-4444-9615-bf168854a75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4"
Nov 03 20:40:41.891337 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:41.891245    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4"
Nov 03 20:40:42.000305 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.000270    5591 generic.go:356] "Generic (PLEG): container finished" podID="9d797881-05fe-4105-9dbe-e7ab7b768982" containerID="0353ac9841442ed0b2b459195ab45ca668367f62a25e9870a28e1a8e48004e72" exitCode=0
Nov 03 20:40:42.000765 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.000315    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"
Nov 03 20:40:42.000765 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.000344    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" event={"ID":"9d797881-05fe-4105-9dbe-e7ab7b768982","Type":"ContainerDied","Data":"0353ac9841442ed0b2b459195ab45ca668367f62a25e9870a28e1a8e48004e72"}
Nov 03 20:40:42.000765 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.000395    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc" event={"ID":"9d797881-05fe-4105-9dbe-e7ab7b768982","Type":"ContainerDied","Data":"c51bb9976878005c8ebe27b1001d6bf38e3d648dd6589a498c1e957228427c72"}
Nov 03 20:40:42.000765 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.000422    5591 scope.go:122] "RemoveContainer" containerID="0353ac9841442ed0b2b459195ab45ca668367f62a25e9870a28e1a8e48004e72"
Nov 03 20:40:42.006594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.005068    5591 generic.go:356] "Generic (PLEG): container finished" podID="665623ee-f9a4-4aae-922b-f229460a9cf2" containerID="036a29cf2b55fe80c230491d6447b63f0f5cabb2fe4ebffa140e25fad7587edf" exitCode=0
Nov 03 20:40:42.006594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.005185    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh" event={"ID":"665623ee-f9a4-4aae-922b-f229460a9cf2","Type":"ContainerDied","Data":"036a29cf2b55fe80c230491d6447b63f0f5cabb2fe4ebffa140e25fad7587edf"}
Nov 03 20:40:42.006594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.005244    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh" event={"ID":"665623ee-f9a4-4aae-922b-f229460a9cf2","Type":"ContainerDied","Data":"9c144af027b1af358d1dd753254f35896800722063b41f840223821306edb1e9"}
Nov 03 20:40:42.006594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.005257    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"
Nov 03 20:40:42.013171 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.013140    5591 generic.go:356] "Generic (PLEG): container finished" podID="8cf0c23f-8862-442a-a487-35f4023789b8" containerID="e3b01a953a5f0a40565252edadff3c23cb9b38a7481930778869d0e03761c66d" exitCode=0
Nov 03 20:40:42.013359 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.013339    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" event={"ID":"8cf0c23f-8862-442a-a487-35f4023789b8","Type":"ContainerDied","Data":"e3b01a953a5f0a40565252edadff3c23cb9b38a7481930778869d0e03761c66d"}
Nov 03 20:40:42.013489 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.013375    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4" event={"ID":"8cf0c23f-8862-442a-a487-35f4023789b8","Type":"ContainerDied","Data":"c0955f6febd1005f9b4623fcd96bdc84bc3d8a8e971d550567df829f7c1ca4c7"}
Nov 03 20:40:42.013489 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.013466    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"
Nov 03 20:40:42.019567 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.019541    5591 scope.go:122] "RemoveContainer" containerID="0353ac9841442ed0b2b459195ab45ca668367f62a25e9870a28e1a8e48004e72"
Nov 03 20:40:42.019940 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:40:42.019910    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0353ac9841442ed0b2b459195ab45ca668367f62a25e9870a28e1a8e48004e72\": container with ID starting with 0353ac9841442ed0b2b459195ab45ca668367f62a25e9870a28e1a8e48004e72 not found: ID does not exist" containerID="0353ac9841442ed0b2b459195ab45ca668367f62a25e9870a28e1a8e48004e72"
Nov 03 20:40:42.020038 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.019952    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0353ac9841442ed0b2b459195ab45ca668367f62a25e9870a28e1a8e48004e72"} err="failed to get container status \"0353ac9841442ed0b2b459195ab45ca668367f62a25e9870a28e1a8e48004e72\": rpc error: code = NotFound desc = could not find container \"0353ac9841442ed0b2b459195ab45ca668367f62a25e9870a28e1a8e48004e72\": container with ID starting with 0353ac9841442ed0b2b459195ab45ca668367f62a25e9870a28e1a8e48004e72 not found: ID does not exist"
Nov 03 20:40:42.020038 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.019975    5591 scope.go:122] "RemoveContainer" containerID="036a29cf2b55fe80c230491d6447b63f0f5cabb2fe4ebffa140e25fad7587edf"
Nov 03 20:40:42.044320 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.044300    5591 scope.go:122] "RemoveContainer" containerID="036a29cf2b55fe80c230491d6447b63f0f5cabb2fe4ebffa140e25fad7587edf"
Nov 03 20:40:42.044625 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:40:42.044603    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"036a29cf2b55fe80c230491d6447b63f0f5cabb2fe4ebffa140e25fad7587edf\": container with ID starting with 036a29cf2b55fe80c230491d6447b63f0f5cabb2fe4ebffa140e25fad7587edf not found: ID does not exist" containerID="036a29cf2b55fe80c230491d6447b63f0f5cabb2fe4ebffa140e25fad7587edf"
Nov 03 20:40:42.044703 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.044634    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"036a29cf2b55fe80c230491d6447b63f0f5cabb2fe4ebffa140e25fad7587edf"} err="failed to get container status \"036a29cf2b55fe80c230491d6447b63f0f5cabb2fe4ebffa140e25fad7587edf\": rpc error: code = NotFound desc = could not find container \"036a29cf2b55fe80c230491d6447b63f0f5cabb2fe4ebffa140e25fad7587edf\": container with ID starting with 036a29cf2b55fe80c230491d6447b63f0f5cabb2fe4ebffa140e25fad7587edf not found: ID does not exist"
Nov 03 20:40:42.044703 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.044653    5591 scope.go:122] "RemoveContainer" containerID="e3b01a953a5f0a40565252edadff3c23cb9b38a7481930778869d0e03761c66d"
Nov 03 20:40:42.080083 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.080052    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"]
Nov 03 20:40:42.085228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.083005    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/distinct-2-w3-9f46e5-kserve-5cd7db5d46-6n8zc"]
Nov 03 20:40:42.093381 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.093098    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"]
Nov 03 20:40:42.093381 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.093273    5591 scope.go:122] "RemoveContainer" containerID="e3b01a953a5f0a40565252edadff3c23cb9b38a7481930778869d0e03761c66d"
Nov 03 20:40:42.093730 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:40:42.093706    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3b01a953a5f0a40565252edadff3c23cb9b38a7481930778869d0e03761c66d\": container with ID starting with e3b01a953a5f0a40565252edadff3c23cb9b38a7481930778869d0e03761c66d not found: ID does not exist" containerID="e3b01a953a5f0a40565252edadff3c23cb9b38a7481930778869d0e03761c66d"
Nov 03 20:40:42.093868 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.093741    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3b01a953a5f0a40565252edadff3c23cb9b38a7481930778869d0e03761c66d"} err="failed to get container status \"e3b01a953a5f0a40565252edadff3c23cb9b38a7481930778869d0e03761c66d\": rpc error: code = NotFound desc = could not find container \"e3b01a953a5f0a40565252edadff3c23cb9b38a7481930778869d0e03761c66d\": container with ID starting with e3b01a953a5f0a40565252edadff3c23cb9b38a7481930778869d0e03761c66d not found: ID does not exist"
Nov 03 20:40:42.101774 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.101745    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/premium-w3-9f46e5-kserve-66b569d97d-g8hq4"]
Nov 03 20:40:42.111959 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.111935    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"]
Nov 03 20:40:42.117558 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.117537    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-9f46e5/simulator-w3-9f46e5-kserve-69bd6bd7ff-jrrwh"]
Nov 03 20:40:42.185132 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:42.185098    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4"]
Nov 03 20:40:42.186635 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:40:42.186602    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7880a3e1_aceb_4444_9615_bf168854a75b.slice/crio-f282b129b983e544a0f7ede8cb59d1c53482c4c74b3c64553289853e3cbd36fe WatchSource:0}: Error finding container f282b129b983e544a0f7ede8cb59d1c53482c4c74b3c64553289853e3cbd36fe: Status 404 returned error can't find the container with id f282b129b983e544a0f7ede8cb59d1c53482c4c74b3c64553289853e3cbd36fe
Nov 03 20:40:43.019628 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:43.019580    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4" event={"ID":"7880a3e1-aceb-4444-9615-bf168854a75b","Type":"ContainerStarted","Data":"7494561d91fab4ed829766c3aa14f32b16e6fc9406d467fe5cb421b3f0e4d01a"}
Nov 03 20:40:43.020131 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:43.019648    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4" event={"ID":"7880a3e1-aceb-4444-9615-bf168854a75b","Type":"ContainerStarted","Data":"f282b129b983e544a0f7ede8cb59d1c53482c4c74b3c64553289853e3cbd36fe"}
Nov 03 20:40:43.037982 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:43.037926    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4" podStartSLOduration=2.037908175 podStartE2EDuration="2.037908175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:43.036939263 +0000 UTC m=+3791.809232274" watchObservedRunningTime="2026-11-03 20:40:43.037908175 +0000 UTC m=+3791.810201194"
Nov 03 20:40:43.352470 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:43.351948    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="665623ee-f9a4-4aae-922b-f229460a9cf2" path="/var/lib/kubelet/pods/665623ee-f9a4-4aae-922b-f229460a9cf2/volumes"
Nov 03 20:40:43.352671 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:43.352637    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cf0c23f-8862-442a-a487-35f4023789b8" path="/var/lib/kubelet/pods/8cf0c23f-8862-442a-a487-35f4023789b8/volumes"
Nov 03 20:40:43.355661 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:43.353321    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d797881-05fe-4105-9dbe-e7ab7b768982" path="/var/lib/kubelet/pods/9d797881-05fe-4105-9dbe-e7ab7b768982/volumes"
Nov 03 20:40:44.031039 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:44.030996    5591 generic.go:356] "Generic (PLEG): container finished" podID="7880a3e1-aceb-4444-9615-bf168854a75b" containerID="7494561d91fab4ed829766c3aa14f32b16e6fc9406d467fe5cb421b3f0e4d01a" exitCode=0
Nov 03 20:40:44.031481 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:44.031060    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4" event={"ID":"7880a3e1-aceb-4444-9615-bf168854a75b","Type":"ContainerDied","Data":"7494561d91fab4ed829766c3aa14f32b16e6fc9406d467fe5cb421b3f0e4d01a"}
Nov 03 20:40:44.864662 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:44.864629    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"]
Nov 03 20:40:44.864928 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:44.864894    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s" podUID="a223486f-1753-4dd1-8fd5-a6c03d664555" containerName="istio-proxy" containerID="cri-o://223d0287fb8ef34d6dd45d94490ec148c5e07cfd2b40f839e25e7de690956c5e" gracePeriod=30
Nov 03 20:40:44.865827 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:44.865432    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"]
Nov 03 20:40:44.865827 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:44.865600    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v" podUID="6d7456ba-061d-4c8a-974c-326c72027557" containerName="istio-proxy" containerID="cri-o://0f19d6863d124322e0a6ffba90bd2d002acc852418bdd319f483f6a8b900a5f4" gracePeriod=30
Nov 03 20:40:45.203406 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:45.203378    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4"
Nov 03 20:40:45.215001 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:45.214974    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7880a3e1-aceb-4444-9615-bf168854a75b-kube-api-access-h5dx9\" (UniqueName: \"kubernetes.io/projected/7880a3e1-aceb-4444-9615-bf168854a75b-kube-api-access-h5dx9\") pod \"7880a3e1-aceb-4444-9615-bf168854a75b\" (UID: \"7880a3e1-aceb-4444-9615-bf168854a75b\") "
Nov 03 20:40:45.215116 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:45.215018    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7880a3e1-aceb-4444-9615-bf168854a75b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7880a3e1-aceb-4444-9615-bf168854a75b-maas-api-openshift-ca\") pod \"7880a3e1-aceb-4444-9615-bf168854a75b\" (UID: \"7880a3e1-aceb-4444-9615-bf168854a75b\") "
Nov 03 20:40:45.215154 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:45.215137    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7880a3e1-aceb-4444-9615-bf168854a75b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7880a3e1-aceb-4444-9615-bf168854a75b-maas-api-xks-ca\") pod \"7880a3e1-aceb-4444-9615-bf168854a75b\" (UID: \"7880a3e1-aceb-4444-9615-bf168854a75b\") "
Nov 03 20:40:45.215484 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:45.215456    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7880a3e1-aceb-4444-9615-bf168854a75b-maas-api-openshift-ca" pod "7880a3e1-aceb-4444-9615-bf168854a75b" (UID: "7880a3e1-aceb-4444-9615-bf168854a75b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:40:45.217358 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:45.217333    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7880a3e1-aceb-4444-9615-bf168854a75b-kube-api-access-h5dx9" pod "7880a3e1-aceb-4444-9615-bf168854a75b" (UID: "7880a3e1-aceb-4444-9615-bf168854a75b"). InnerVolumeSpecName "kube-api-access-h5dx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:45.217481 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:45.217441    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7880a3e1-aceb-4444-9615-bf168854a75b-maas-api-xks-ca" pod "7880a3e1-aceb-4444-9615-bf168854a75b" (UID: "7880a3e1-aceb-4444-9615-bf168854a75b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:40:45.316260 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:45.316197    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5dx9\" (UniqueName: \"kubernetes.io/projected/7880a3e1-aceb-4444-9615-bf168854a75b-kube-api-access-h5dx9\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:45.316260 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:45.316252    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7880a3e1-aceb-4444-9615-bf168854a75b-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:45.316260 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:45.316263    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7880a3e1-aceb-4444-9615-bf168854a75b-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:46.083753 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:46.083719    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4"
Nov 03 20:40:46.083969 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:46.083759    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4" event={"ID":"7880a3e1-aceb-4444-9615-bf168854a75b","Type":"ContainerDied","Data":"f282b129b983e544a0f7ede8cb59d1c53482c4c74b3c64553289853e3cbd36fe"}
Nov 03 20:40:46.083969 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:46.083816    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f282b129b983e544a0f7ede8cb59d1c53482c4c74b3c64553289853e3cbd36fe"
Nov 03 20:40:50.109698 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.109660    5591 generic.go:356] "Generic (PLEG): container finished" podID="a223486f-1753-4dd1-8fd5-a6c03d664555" containerID="223d0287fb8ef34d6dd45d94490ec148c5e07cfd2b40f839e25e7de690956c5e" exitCode=0
Nov 03 20:40:50.110127 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.109768    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s" event={"ID":"a223486f-1753-4dd1-8fd5-a6c03d664555","Type":"ContainerDied","Data":"223d0287fb8ef34d6dd45d94490ec148c5e07cfd2b40f839e25e7de690956c5e"}
Nov 03 20:40:50.111619 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.111592    5591 generic.go:356] "Generic (PLEG): container finished" podID="6d7456ba-061d-4c8a-974c-326c72027557" containerID="0f19d6863d124322e0a6ffba90bd2d002acc852418bdd319f483f6a8b900a5f4" exitCode=0
Nov 03 20:40:50.111770 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.111752    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v" event={"ID":"6d7456ba-061d-4c8a-974c-326c72027557","Type":"ContainerDied","Data":"0f19d6863d124322e0a6ffba90bd2d002acc852418bdd319f483f6a8b900a5f4"}
Nov 03 20:40:50.111860 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.111800    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v" event={"ID":"6d7456ba-061d-4c8a-974c-326c72027557","Type":"ContainerDied","Data":"dc34fdacf9b8533e3f02bc8e6ffed2ff9e66000c1d75f814e8f0a66be76623d1"}
Nov 03 20:40:50.111860 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.111824    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc34fdacf9b8533e3f02bc8e6ffed2ff9e66000c1d75f814e8f0a66be76623d1"
Nov 03 20:40:50.129246 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.129226    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 20:40:50.156297 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.156270    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 20:40:50.166349 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.166322    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d7456ba-061d-4c8a-974c-326c72027557-kube-api-access-qk6rp\" (UniqueName: \"kubernetes.io/projected/6d7456ba-061d-4c8a-974c-326c72027557-kube-api-access-qk6rp\") pod \"6d7456ba-061d-4c8a-974c-326c72027557\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") "
Nov 03 20:40:50.166502 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.166411    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-credential-socket\") pod \"a223486f-1753-4dd1-8fd5-a6c03d664555\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") "
Nov 03 20:40:50.166502 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.166444    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-token\" (UniqueName: \"kubernetes.io/projected/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-token\") pod \"a223486f-1753-4dd1-8fd5-a6c03d664555\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") "
Nov 03 20:40:50.166502 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.166482    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-credential-socket\") pod \"6d7456ba-061d-4c8a-974c-326c72027557\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") "
Nov 03 20:40:50.166688 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.166513    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-istio-envoy\") pod \"6d7456ba-061d-4c8a-974c-326c72027557\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") "
Nov 03 20:40:50.166688 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.166540    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a223486f-1753-4dd1-8fd5-a6c03d664555-kube-api-access-wfrcq\" (UniqueName: \"kubernetes.io/projected/a223486f-1753-4dd1-8fd5-a6c03d664555-kube-api-access-wfrcq\") pod \"a223486f-1753-4dd1-8fd5-a6c03d664555\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") "
Nov 03 20:40:50.166688 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.166585    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-podinfo\") pod \"a223486f-1753-4dd1-8fd5-a6c03d664555\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") "
Nov 03 20:40:50.166688 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.166661    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-workload-socket\") pod \"6d7456ba-061d-4c8a-974c-326c72027557\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") "
Nov 03 20:40:50.166917 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.166689    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a223486f-1753-4dd1-8fd5-a6c03d664555-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a223486f-1753-4dd1-8fd5-a6c03d664555-istiod-ca-cert\") pod \"a223486f-1753-4dd1-8fd5-a6c03d664555\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") "
Nov 03 20:40:50.166917 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.166698    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-credential-socket" pod "a223486f-1753-4dd1-8fd5-a6c03d664555" (UID: "a223486f-1753-4dd1-8fd5-a6c03d664555"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:50.166917 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.166719    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-istio-data\") pod \"6d7456ba-061d-4c8a-974c-326c72027557\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") "
Nov 03 20:40:50.166917 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.166757    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d7456ba-061d-4c8a-974c-326c72027557-istio-token\" (UniqueName: \"kubernetes.io/projected/6d7456ba-061d-4c8a-974c-326c72027557-istio-token\") pod \"6d7456ba-061d-4c8a-974c-326c72027557\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") "
Nov 03 20:40:50.166917 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.166787    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-envoy\") pod \"a223486f-1753-4dd1-8fd5-a6c03d664555\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") "
Nov 03 20:40:50.166917 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.166869    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d7456ba-061d-4c8a-974c-326c72027557-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d7456ba-061d-4c8a-974c-326c72027557-istiod-ca-cert\") pod \"6d7456ba-061d-4c8a-974c-326c72027557\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") "
Nov 03 20:40:50.166917 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.166902    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-workload-certs\") pod \"6d7456ba-061d-4c8a-974c-326c72027557\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") "
Nov 03 20:40:50.167354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.166945    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-workload-socket\") pod \"a223486f-1753-4dd1-8fd5-a6c03d664555\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") "
Nov 03 20:40:50.167354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.166974    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-data\") pod \"a223486f-1753-4dd1-8fd5-a6c03d664555\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") "
Nov 03 20:40:50.167354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.167016    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d7456ba-061d-4c8a-974c-326c72027557-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d7456ba-061d-4c8a-974c-326c72027557-istio-podinfo\") pod \"6d7456ba-061d-4c8a-974c-326c72027557\" (UID: \"6d7456ba-061d-4c8a-974c-326c72027557\") "
Nov 03 20:40:50.167354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.167045    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-workload-certs\") pod \"a223486f-1753-4dd1-8fd5-a6c03d664555\" (UID: \"a223486f-1753-4dd1-8fd5-a6c03d664555\") "
Nov 03 20:40:50.167566 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.167394    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-workload-socket" pod "6d7456ba-061d-4c8a-974c-326c72027557" (UID: "6d7456ba-061d-4c8a-974c-326c72027557"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:50.167566 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.167490    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-credential-socket" pod "6d7456ba-061d-4c8a-974c-326c72027557" (UID: "6d7456ba-061d-4c8a-974c-326c72027557"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:50.167566 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.167549    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:50.167746 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.167556    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-workload-socket" pod "a223486f-1753-4dd1-8fd5-a6c03d664555" (UID: "a223486f-1753-4dd1-8fd5-a6c03d664555"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:50.167746 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.167570    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:50.167746 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.167641    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-workload-certs" pod "a223486f-1753-4dd1-8fd5-a6c03d664555" (UID: "a223486f-1753-4dd1-8fd5-a6c03d664555"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:50.167916 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.167789    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-data" pod "a223486f-1753-4dd1-8fd5-a6c03d664555" (UID: "a223486f-1753-4dd1-8fd5-a6c03d664555"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:50.167916 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.167892    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d7456ba-061d-4c8a-974c-326c72027557-istiod-ca-cert" pod "6d7456ba-061d-4c8a-974c-326c72027557" (UID: "6d7456ba-061d-4c8a-974c-326c72027557"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:40:50.168271 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.168143    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-istio-data" pod "6d7456ba-061d-4c8a-974c-326c72027557" (UID: "6d7456ba-061d-4c8a-974c-326c72027557"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:50.168271 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.168239    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-workload-certs" pod "6d7456ba-061d-4c8a-974c-326c72027557" (UID: "6d7456ba-061d-4c8a-974c-326c72027557"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:50.168645 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.168619    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a223486f-1753-4dd1-8fd5-a6c03d664555-istiod-ca-cert" pod "a223486f-1753-4dd1-8fd5-a6c03d664555" (UID: "a223486f-1753-4dd1-8fd5-a6c03d664555"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:40:50.170497 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.170464    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a223486f-1753-4dd1-8fd5-a6c03d664555-kube-api-access-wfrcq" pod "a223486f-1753-4dd1-8fd5-a6c03d664555" (UID: "a223486f-1753-4dd1-8fd5-a6c03d664555"). InnerVolumeSpecName "kube-api-access-wfrcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:50.170613 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.170550    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d7456ba-061d-4c8a-974c-326c72027557-istio-podinfo" pod "6d7456ba-061d-4c8a-974c-326c72027557" (UID: "6d7456ba-061d-4c8a-974c-326c72027557"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:40:50.170898 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.170870    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-podinfo" pod "a223486f-1753-4dd1-8fd5-a6c03d664555" (UID: "a223486f-1753-4dd1-8fd5-a6c03d664555"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:40:50.171005 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.170944    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d7456ba-061d-4c8a-974c-326c72027557-istio-token" pod "6d7456ba-061d-4c8a-974c-326c72027557" (UID: "6d7456ba-061d-4c8a-974c-326c72027557"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:50.171005 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.170977    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-istio-envoy" pod "6d7456ba-061d-4c8a-974c-326c72027557" (UID: "6d7456ba-061d-4c8a-974c-326c72027557"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:50.171005 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.170984    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-token" pod "a223486f-1753-4dd1-8fd5-a6c03d664555" (UID: "a223486f-1753-4dd1-8fd5-a6c03d664555"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:50.171531 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.171506    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-envoy" pod "a223486f-1753-4dd1-8fd5-a6c03d664555" (UID: "a223486f-1753-4dd1-8fd5-a6c03d664555"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:50.172496 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.172469    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d7456ba-061d-4c8a-974c-326c72027557-kube-api-access-qk6rp" pod "6d7456ba-061d-4c8a-974c-326c72027557" (UID: "6d7456ba-061d-4c8a-974c-326c72027557"). InnerVolumeSpecName "kube-api-access-qk6rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:50.268939 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.268896    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a223486f-1753-4dd1-8fd5-a6c03d664555-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:50.268939 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.268926    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:50.268939 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.268935    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d7456ba-061d-4c8a-974c-326c72027557-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:50.268939 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.268944    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:50.269237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.268952    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d7456ba-061d-4c8a-974c-326c72027557-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:50.269237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.268961    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:50.269237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.268971    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:50.269237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.268979    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:50.269237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.268986    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d7456ba-061d-4c8a-974c-326c72027557-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:50.269237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.268994    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a223486f-1753-4dd1-8fd5-a6c03d664555-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:50.269237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.269002    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk6rp\" (UniqueName: \"kubernetes.io/projected/6d7456ba-061d-4c8a-974c-326c72027557-kube-api-access-qk6rp\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:50.269237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.269013    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:50.269237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.269022    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:50.269237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.269030    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d7456ba-061d-4c8a-974c-326c72027557-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:50.269237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.269038    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfrcq\" (UniqueName: \"kubernetes.io/projected/a223486f-1753-4dd1-8fd5-a6c03d664555-kube-api-access-wfrcq\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:50.269237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:50.269046    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a223486f-1753-4dd1-8fd5-a6c03d664555-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:51.121465 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:51.121433    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"
Nov 03 20:40:51.121925 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:51.121466    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s" event={"ID":"a223486f-1753-4dd1-8fd5-a6c03d664555","Type":"ContainerDied","Data":"650b92723bc563b0e89f2af94921237342fa083e50a95a450fbb40bc25ad0e4c"}
Nov 03 20:40:51.121925 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:51.121534    5591 scope.go:122] "RemoveContainer" containerID="223d0287fb8ef34d6dd45d94490ec148c5e07cfd2b40f839e25e7de690956c5e"
Nov 03 20:40:51.121925 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:51.121756    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"
Nov 03 20:40:51.148227 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:51.148168    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"]
Nov 03 20:40:51.157538 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:51.157512    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-dh77s"]
Nov 03 20:40:51.171853 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:51.171833    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"]
Nov 03 20:40:51.178310 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:51.178276    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-32e482-openshift-default-7cfdc6fb4c-76h6v"]
Nov 03 20:40:51.364847 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:51.364808    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d7456ba-061d-4c8a-974c-326c72027557" path="/var/lib/kubelet/pods/6d7456ba-061d-4c8a-974c-326c72027557/volumes"
Nov 03 20:40:51.365638 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:51.365619    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a223486f-1753-4dd1-8fd5-a6c03d664555" path="/var/lib/kubelet/pods/a223486f-1753-4dd1-8fd5-a6c03d664555/volumes"
Nov 03 20:40:51.919654 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:51.919619    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:40:51.972445 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:51.972412    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j"]
Nov 03 20:40:51.972693 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:51.972662    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j" podUID="85f37dfd-b7be-4333-a7c2-489edff95788" containerName="maas-api" containerID="cri-o://d8f914ec65b8ca458c7f4cc3b39830cdd403596c79e780bd4b3ac95489ec7cbb" gracePeriod=30
Nov 03 20:40:52.067328 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:52.067287    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4"]
Nov 03 20:40:52.070440 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:52.070408    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2b5703-rwhv4"]
Nov 03 20:40:52.131266 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:52.131234    5591 generic.go:356] "Generic (PLEG): container finished" podID="85f37dfd-b7be-4333-a7c2-489edff95788" containerID="d8f914ec65b8ca458c7f4cc3b39830cdd403596c79e780bd4b3ac95489ec7cbb" exitCode=0
Nov 03 20:40:52.131683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:52.131288    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j" event={"ID":"85f37dfd-b7be-4333-a7c2-489edff95788","Type":"ContainerDied","Data":"d8f914ec65b8ca458c7f4cc3b39830cdd403596c79e780bd4b3ac95489ec7cbb"}
Nov 03 20:40:52.293072 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:52.293039    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j"
Nov 03 20:40:52.392057 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:52.390596    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85f37dfd-b7be-4333-a7c2-489edff95788-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f37dfd-b7be-4333-a7c2-489edff95788-openshift-service-ca\") pod \"85f37dfd-b7be-4333-a7c2-489edff95788\" (UID: \"85f37dfd-b7be-4333-a7c2-489edff95788\") "
Nov 03 20:40:52.392057 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:52.390655    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f37dfd-b7be-4333-a7c2-489edff95788-metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f37dfd-b7be-4333-a7c2-489edff95788-metrics-certs\") pod \"85f37dfd-b7be-4333-a7c2-489edff95788\" (UID: \"85f37dfd-b7be-4333-a7c2-489edff95788\") "
Nov 03 20:40:52.392057 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:52.390734    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f37dfd-b7be-4333-a7c2-489edff95788-kube-api-access-qcjq2\" (UniqueName: \"kubernetes.io/projected/85f37dfd-b7be-4333-a7c2-489edff95788-kube-api-access-qcjq2\") pod \"85f37dfd-b7be-4333-a7c2-489edff95788\" (UID: \"85f37dfd-b7be-4333-a7c2-489edff95788\") "
Nov 03 20:40:52.392057 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:52.390810    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f37dfd-b7be-4333-a7c2-489edff95788-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f37dfd-b7be-4333-a7c2-489edff95788-maas-api-tls\") pod \"85f37dfd-b7be-4333-a7c2-489edff95788\" (UID: \"85f37dfd-b7be-4333-a7c2-489edff95788\") "
Nov 03 20:40:52.392057 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:52.390995    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f37dfd-b7be-4333-a7c2-489edff95788-openshift-service-ca" pod "85f37dfd-b7be-4333-a7c2-489edff95788" (UID: "85f37dfd-b7be-4333-a7c2-489edff95788"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:40:52.392057 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:52.391323    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f37dfd-b7be-4333-a7c2-489edff95788-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:52.392057 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:52.390600    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"]
Nov 03 20:40:52.395389 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:52.395356    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f37dfd-b7be-4333-a7c2-489edff95788-metrics-certs" pod "85f37dfd-b7be-4333-a7c2-489edff95788" (UID: "85f37dfd-b7be-4333-a7c2-489edff95788"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:40:52.395900 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:52.395874    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"]
Nov 03 20:40:52.397533 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:52.397497    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7" podUID="73a74bc3-6a5f-46be-8094-74d859931e19" containerName="istio-proxy" containerID="cri-o://3d655a9905f2f1c41cdfb10e9f97ca1901907d541062e37be515ecb0ce36be60" gracePeriod=30
Nov 03 20:40:52.398710 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:52.398671    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f37dfd-b7be-4333-a7c2-489edff95788-kube-api-access-qcjq2" pod "85f37dfd-b7be-4333-a7c2-489edff95788" (UID: "85f37dfd-b7be-4333-a7c2-489edff95788"). InnerVolumeSpecName "kube-api-access-qcjq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:52.399243 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:52.399220    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f37dfd-b7be-4333-a7c2-489edff95788-maas-api-tls" pod "85f37dfd-b7be-4333-a7c2-489edff95788" (UID: "85f37dfd-b7be-4333-a7c2-489edff95788"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:40:52.401191 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:52.401125    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks" podUID="328b01ad-1c34-473f-8e11-5e131775edc6" containerName="istio-proxy" containerID="cri-o://314b84220b6acd5ed6e19f316cdcc073adf405b4a3ae04e4077b13a3662a53c8" gracePeriod=30
Nov 03 20:40:52.492284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:52.492245    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f37dfd-b7be-4333-a7c2-489edff95788-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:52.492284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:52.492274    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f37dfd-b7be-4333-a7c2-489edff95788-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:52.492284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:52.492286    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcjq2\" (UniqueName: \"kubernetes.io/projected/85f37dfd-b7be-4333-a7c2-489edff95788-kube-api-access-qcjq2\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:53.140069 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:53.140032    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j"
Nov 03 20:40:53.140663 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:53.140065    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j" event={"ID":"85f37dfd-b7be-4333-a7c2-489edff95788","Type":"ContainerDied","Data":"4b8aab70f1b8e84627a7bd16b6b24527c73f4b5c40c13c21f4d99a1ae95aa086"}
Nov 03 20:40:53.140663 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:53.140141    5591 scope.go:122] "RemoveContainer" containerID="d8f914ec65b8ca458c7f4cc3b39830cdd403596c79e780bd4b3ac95489ec7cbb"
Nov 03 20:40:53.163816 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:53.163773    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j"]
Nov 03 20:40:53.165321 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:53.165298    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-9f46e5-d88498d5d-6cs8j"]
Nov 03 20:40:53.348069 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:53.348037    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7880a3e1-aceb-4444-9615-bf168854a75b" path="/var/lib/kubelet/pods/7880a3e1-aceb-4444-9615-bf168854a75b/volumes"
Nov 03 20:40:53.348491 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:53.348477    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f37dfd-b7be-4333-a7c2-489edff95788" path="/var/lib/kubelet/pods/85f37dfd-b7be-4333-a7c2-489edff95788/volumes"
Nov 03 20:40:56.696600 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:56.696566    5591 patch_prober.go:29] interesting pod/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:40:56.696997 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:56.696625    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7" podUID="73a74bc3-6a5f-46be-8094-74d859931e19" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:40:57.671801 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.671734    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 20:40:57.704236 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.703372    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:57.735039 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.735005    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:40:57.748381 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.748335    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-workload-certs\") pod \"73a74bc3-6a5f-46be-8094-74d859931e19\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") "
Nov 03 20:40:57.748558 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.748444    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73a74bc3-6a5f-46be-8094-74d859931e19-istio-token\" (UniqueName: \"kubernetes.io/projected/73a74bc3-6a5f-46be-8094-74d859931e19-istio-token\") pod \"73a74bc3-6a5f-46be-8094-74d859931e19\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") "
Nov 03 20:40:57.748558 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.748481    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73a74bc3-6a5f-46be-8094-74d859931e19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a74bc3-6a5f-46be-8094-74d859931e19-istio-podinfo\") pod \"73a74bc3-6a5f-46be-8094-74d859931e19\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") "
Nov 03 20:40:57.748558 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.748512    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-credential-socket\") pod \"73a74bc3-6a5f-46be-8094-74d859931e19\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") "
Nov 03 20:40:57.748699 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.748617    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-workload-certs" pod "73a74bc3-6a5f-46be-8094-74d859931e19" (UID: "73a74bc3-6a5f-46be-8094-74d859931e19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:57.748699 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.748666    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73a74bc3-6a5f-46be-8094-74d859931e19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a74bc3-6a5f-46be-8094-74d859931e19-istiod-ca-cert\") pod \"73a74bc3-6a5f-46be-8094-74d859931e19\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") "
Nov 03 20:40:57.748806 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.748702    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/328b01ad-1c34-473f-8e11-5e131775edc6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/328b01ad-1c34-473f-8e11-5e131775edc6-istio-podinfo\") pod \"328b01ad-1c34-473f-8e11-5e131775edc6\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") "
Nov 03 20:40:57.748878 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.748808    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-workload-socket\") pod \"328b01ad-1c34-473f-8e11-5e131775edc6\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") "
Nov 03 20:40:57.748878 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.748834    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-credential-socket" pod "73a74bc3-6a5f-46be-8094-74d859931e19" (UID: "73a74bc3-6a5f-46be-8094-74d859931e19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:57.748878 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.748838    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-workload-socket\") pod \"73a74bc3-6a5f-46be-8094-74d859931e19\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") "
Nov 03 20:40:57.749043 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.748932    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-workload-certs\") pod \"328b01ad-1c34-473f-8e11-5e131775edc6\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") "
Nov 03 20:40:57.749043 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.748990    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-istio-envoy\") pod \"73a74bc3-6a5f-46be-8094-74d859931e19\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") "
Nov 03 20:40:57.749043 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.749000    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-workload-socket" pod "73a74bc3-6a5f-46be-8094-74d859931e19" (UID: "73a74bc3-6a5f-46be-8094-74d859931e19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:57.749222 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.749045    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/328b01ad-1c34-473f-8e11-5e131775edc6-istio-token\" (UniqueName: \"kubernetes.io/projected/328b01ad-1c34-473f-8e11-5e131775edc6-istio-token\") pod \"328b01ad-1c34-473f-8e11-5e131775edc6\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") "
Nov 03 20:40:57.749222 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.749077    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-istio-data\") pod \"328b01ad-1c34-473f-8e11-5e131775edc6\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") "
Nov 03 20:40:57.749222 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.749115    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-istio-envoy\") pod \"328b01ad-1c34-473f-8e11-5e131775edc6\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") "
Nov 03 20:40:57.749222 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.749151    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73a74bc3-6a5f-46be-8094-74d859931e19-kube-api-access-k4lfh\" (UniqueName: \"kubernetes.io/projected/73a74bc3-6a5f-46be-8094-74d859931e19-kube-api-access-k4lfh\") pod \"73a74bc3-6a5f-46be-8094-74d859931e19\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") "
Nov 03 20:40:57.749222 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.749187    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/328b01ad-1c34-473f-8e11-5e131775edc6-kube-api-access-6m57s\" (UniqueName: \"kubernetes.io/projected/328b01ad-1c34-473f-8e11-5e131775edc6-kube-api-access-6m57s\") pod \"328b01ad-1c34-473f-8e11-5e131775edc6\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") "
Nov 03 20:40:57.749513 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.749177    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-workload-socket" pod "328b01ad-1c34-473f-8e11-5e131775edc6" (UID: "328b01ad-1c34-473f-8e11-5e131775edc6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:57.749513 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.749264    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/328b01ad-1c34-473f-8e11-5e131775edc6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/328b01ad-1c34-473f-8e11-5e131775edc6-istiod-ca-cert\") pod \"328b01ad-1c34-473f-8e11-5e131775edc6\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") "
Nov 03 20:40:57.749513 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.749294    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-istio-data\") pod \"73a74bc3-6a5f-46be-8094-74d859931e19\" (UID: \"73a74bc3-6a5f-46be-8094-74d859931e19\") "
Nov 03 20:40:57.749513 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.749326    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-credential-socket\") pod \"328b01ad-1c34-473f-8e11-5e131775edc6\" (UID: \"328b01ad-1c34-473f-8e11-5e131775edc6\") "
Nov 03 20:40:57.749513 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.749435    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-workload-certs" pod "328b01ad-1c34-473f-8e11-5e131775edc6" (UID: "328b01ad-1c34-473f-8e11-5e131775edc6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:57.749777 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.749660    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-credential-socket" pod "328b01ad-1c34-473f-8e11-5e131775edc6" (UID: "328b01ad-1c34-473f-8e11-5e131775edc6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:57.749836 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.749790    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:57.749836 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.749816    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:57.749836 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.749833    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:57.750002 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.749837    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73a74bc3-6a5f-46be-8094-74d859931e19-istiod-ca-cert" pod "73a74bc3-6a5f-46be-8094-74d859931e19" (UID: "73a74bc3-6a5f-46be-8094-74d859931e19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:40:57.750002 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.749847    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:57.750002 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.749973    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:57.750002 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.749993    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:57.750311 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.750032    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-istio-data" pod "73a74bc3-6a5f-46be-8094-74d859931e19" (UID: "73a74bc3-6a5f-46be-8094-74d859931e19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:57.750311 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.750030    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/328b01ad-1c34-473f-8e11-5e131775edc6-istiod-ca-cert" pod "328b01ad-1c34-473f-8e11-5e131775edc6" (UID: "328b01ad-1c34-473f-8e11-5e131775edc6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:40:57.750484 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.750356    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-istio-data" pod "328b01ad-1c34-473f-8e11-5e131775edc6" (UID: "328b01ad-1c34-473f-8e11-5e131775edc6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:57.753194 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.753093    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73a74bc3-6a5f-46be-8094-74d859931e19-istio-podinfo" pod "73a74bc3-6a5f-46be-8094-74d859931e19" (UID: "73a74bc3-6a5f-46be-8094-74d859931e19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:40:57.753194 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.753157    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-istio-envoy" pod "328b01ad-1c34-473f-8e11-5e131775edc6" (UID: "328b01ad-1c34-473f-8e11-5e131775edc6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:57.753426 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.753294    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-istio-envoy" pod "73a74bc3-6a5f-46be-8094-74d859931e19" (UID: "73a74bc3-6a5f-46be-8094-74d859931e19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:40:57.753426 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.753369    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/328b01ad-1c34-473f-8e11-5e131775edc6-istio-podinfo" pod "328b01ad-1c34-473f-8e11-5e131775edc6" (UID: "328b01ad-1c34-473f-8e11-5e131775edc6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:40:57.753745 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.753701    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/328b01ad-1c34-473f-8e11-5e131775edc6-istio-token" pod "328b01ad-1c34-473f-8e11-5e131775edc6" (UID: "328b01ad-1c34-473f-8e11-5e131775edc6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:57.753909 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.753853    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a74bc3-6a5f-46be-8094-74d859931e19-kube-api-access-k4lfh" pod "73a74bc3-6a5f-46be-8094-74d859931e19" (UID: "73a74bc3-6a5f-46be-8094-74d859931e19"). InnerVolumeSpecName "kube-api-access-k4lfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:57.754144 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.754107    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a74bc3-6a5f-46be-8094-74d859931e19-istio-token" pod "73a74bc3-6a5f-46be-8094-74d859931e19" (UID: "73a74bc3-6a5f-46be-8094-74d859931e19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:57.754964 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.754940    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/328b01ad-1c34-473f-8e11-5e131775edc6-kube-api-access-6m57s" pod "328b01ad-1c34-473f-8e11-5e131775edc6" (UID: "328b01ad-1c34-473f-8e11-5e131775edc6"). InnerVolumeSpecName "kube-api-access-6m57s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:40:57.850687 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.850651    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:57.850687 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.850679    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/328b01ad-1c34-473f-8e11-5e131775edc6-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:57.850687 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.850688    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:57.850687 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.850696    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/328b01ad-1c34-473f-8e11-5e131775edc6-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:57.851073 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.850705    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4lfh\" (UniqueName: \"kubernetes.io/projected/73a74bc3-6a5f-46be-8094-74d859931e19-kube-api-access-k4lfh\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:57.851073 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.850714    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m57s\" (UniqueName: \"kubernetes.io/projected/328b01ad-1c34-473f-8e11-5e131775edc6-kube-api-access-6m57s\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:57.851073 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.850722    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/328b01ad-1c34-473f-8e11-5e131775edc6-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:57.851073 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.850732    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a74bc3-6a5f-46be-8094-74d859931e19-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:57.851073 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.850740    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a74bc3-6a5f-46be-8094-74d859931e19-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:57.851073 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.850748    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a74bc3-6a5f-46be-8094-74d859931e19-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:57.851073 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.850756    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a74bc3-6a5f-46be-8094-74d859931e19-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:57.851073 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:57.850763    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/328b01ad-1c34-473f-8e11-5e131775edc6-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:40:58.170718 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:58.170676    5591 generic.go:356] "Generic (PLEG): container finished" podID="328b01ad-1c34-473f-8e11-5e131775edc6" containerID="314b84220b6acd5ed6e19f316cdcc073adf405b4a3ae04e4077b13a3662a53c8" exitCode=0
Nov 03 20:40:58.170909 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:58.170729    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"
Nov 03 20:40:58.170909 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:58.170765    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks" event={"ID":"328b01ad-1c34-473f-8e11-5e131775edc6","Type":"ContainerDied","Data":"314b84220b6acd5ed6e19f316cdcc073adf405b4a3ae04e4077b13a3662a53c8"}
Nov 03 20:40:58.170909 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:58.170819    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks" event={"ID":"328b01ad-1c34-473f-8e11-5e131775edc6","Type":"ContainerDied","Data":"8db202938092fd91245260243a79e728d1f0682dd718453efaae0e3ed1e67637"}
Nov 03 20:40:58.170909 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:58.170850    5591 scope.go:122] "RemoveContainer" containerID="314b84220b6acd5ed6e19f316cdcc073adf405b4a3ae04e4077b13a3662a53c8"
Nov 03 20:40:58.172391 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:58.172313    5591 generic.go:356] "Generic (PLEG): container finished" podID="73a74bc3-6a5f-46be-8094-74d859931e19" containerID="3d655a9905f2f1c41cdfb10e9f97ca1901907d541062e37be515ecb0ce36be60" exitCode=0
Nov 03 20:40:58.172391 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:58.172352    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"
Nov 03 20:40:58.172590 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:58.172528    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7" event={"ID":"73a74bc3-6a5f-46be-8094-74d859931e19","Type":"ContainerDied","Data":"3d655a9905f2f1c41cdfb10e9f97ca1901907d541062e37be515ecb0ce36be60"}
Nov 03 20:40:58.172652 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:58.172610    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7" event={"ID":"73a74bc3-6a5f-46be-8094-74d859931e19","Type":"ContainerDied","Data":"cf48637d75b977083841d6a7dc77235632d94f9bb2147633ba4492c234792c26"}
Nov 03 20:40:58.181086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:58.181068    5591 scope.go:122] "RemoveContainer" containerID="314b84220b6acd5ed6e19f316cdcc073adf405b4a3ae04e4077b13a3662a53c8"
Nov 03 20:40:58.181435 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:40:58.181390    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"314b84220b6acd5ed6e19f316cdcc073adf405b4a3ae04e4077b13a3662a53c8\": container with ID starting with 314b84220b6acd5ed6e19f316cdcc073adf405b4a3ae04e4077b13a3662a53c8 not found: ID does not exist" containerID="314b84220b6acd5ed6e19f316cdcc073adf405b4a3ae04e4077b13a3662a53c8"
Nov 03 20:40:58.181435 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:58.181426    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314b84220b6acd5ed6e19f316cdcc073adf405b4a3ae04e4077b13a3662a53c8"} err="failed to get container status \"314b84220b6acd5ed6e19f316cdcc073adf405b4a3ae04e4077b13a3662a53c8\": rpc error: code = NotFound desc = could not find container \"314b84220b6acd5ed6e19f316cdcc073adf405b4a3ae04e4077b13a3662a53c8\": container with ID starting with 314b84220b6acd5ed6e19f316cdcc073adf405b4a3ae04e4077b13a3662a53c8 not found: ID does not exist"
Nov 03 20:40:58.181574 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:58.181445    5591 scope.go:122] "RemoveContainer" containerID="3d655a9905f2f1c41cdfb10e9f97ca1901907d541062e37be515ecb0ce36be60"
Nov 03 20:40:58.190889 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:58.190868    5591 scope.go:122] "RemoveContainer" containerID="3d655a9905f2f1c41cdfb10e9f97ca1901907d541062e37be515ecb0ce36be60"
Nov 03 20:40:58.191142 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:40:58.191125    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d655a9905f2f1c41cdfb10e9f97ca1901907d541062e37be515ecb0ce36be60\": container with ID starting with 3d655a9905f2f1c41cdfb10e9f97ca1901907d541062e37be515ecb0ce36be60 not found: ID does not exist" containerID="3d655a9905f2f1c41cdfb10e9f97ca1901907d541062e37be515ecb0ce36be60"
Nov 03 20:40:58.191219 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:58.191148    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d655a9905f2f1c41cdfb10e9f97ca1901907d541062e37be515ecb0ce36be60"} err="failed to get container status \"3d655a9905f2f1c41cdfb10e9f97ca1901907d541062e37be515ecb0ce36be60\": rpc error: code = NotFound desc = could not find container \"3d655a9905f2f1c41cdfb10e9f97ca1901907d541062e37be515ecb0ce36be60\": container with ID starting with 3d655a9905f2f1c41cdfb10e9f97ca1901907d541062e37be515ecb0ce36be60 not found: ID does not exist"
Nov 03 20:40:58.197038 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:58.197015    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"]
Nov 03 20:40:58.201275 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:58.201254    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-2lzks"]
Nov 03 20:40:58.213284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:58.213261    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"]
Nov 03 20:40:58.217659 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:58.217635    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b5703-openshift-default-779f8fb9b7-5c6w7"]
Nov 03 20:40:58.723711 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:40:58.723593    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:40:59.348788 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:59.348754    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="328b01ad-1c34-473f-8e11-5e131775edc6" path="/var/lib/kubelet/pods/328b01ad-1c34-473f-8e11-5e131775edc6/volumes"
Nov 03 20:40:59.349256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:40:59.349241    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73a74bc3-6a5f-46be-8094-74d859931e19" path="/var/lib/kubelet/pods/73a74bc3-6a5f-46be-8094-74d859931e19/volumes"
Nov 03 20:41:03.293666 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.292678    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"]
Nov 03 20:41:03.294194 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.293893    5591 cpu_manager.go:414] "containerMap: removing container" podUID="73a74bc3-6a5f-46be-8094-74d859931e19" containerName="istio-proxy"
Nov 03 20:41:03.294194 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.293917    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73a74bc3-6a5f-46be-8094-74d859931e19" containerName="istio-proxy"
Nov 03 20:41:03.294194 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.293967    5591 cpu_manager.go:414] "containerMap: removing container" podUID="6d7456ba-061d-4c8a-974c-326c72027557" containerName="istio-proxy"
Nov 03 20:41:03.294194 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.293974    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d7456ba-061d-4c8a-974c-326c72027557" containerName="istio-proxy"
Nov 03 20:41:03.294194 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.294004    5591 cpu_manager.go:414] "containerMap: removing container" podUID="85f37dfd-b7be-4333-a7c2-489edff95788" containerName="maas-api"
Nov 03 20:41:03.294194 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.294011    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f37dfd-b7be-4333-a7c2-489edff95788" containerName="maas-api"
Nov 03 20:41:03.294194 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.294058    5591 cpu_manager.go:414] "containerMap: removing container" podUID="328b01ad-1c34-473f-8e11-5e131775edc6" containerName="istio-proxy"
Nov 03 20:41:03.294194 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.294066    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="328b01ad-1c34-473f-8e11-5e131775edc6" containerName="istio-proxy"
Nov 03 20:41:03.294194 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.294101    5591 cpu_manager.go:414] "containerMap: removing container" podUID="a223486f-1753-4dd1-8fd5-a6c03d664555" containerName="istio-proxy"
Nov 03 20:41:03.294194 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.294108    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a223486f-1753-4dd1-8fd5-a6c03d664555" containerName="istio-proxy"
Nov 03 20:41:03.294194 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.294118    5591 cpu_manager.go:414] "containerMap: removing container" podUID="7880a3e1-aceb-4444-9615-bf168854a75b" containerName="revoke-keys"
Nov 03 20:41:03.294194 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.294125    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7880a3e1-aceb-4444-9615-bf168854a75b" containerName="revoke-keys"
Nov 03 20:41:03.295480 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.294345    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d7456ba-061d-4c8a-974c-326c72027557" containerName="istio-proxy"
Nov 03 20:41:03.295480 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.294366    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f37dfd-b7be-4333-a7c2-489edff95788" containerName="maas-api"
Nov 03 20:41:03.295480 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.294474    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="328b01ad-1c34-473f-8e11-5e131775edc6" containerName="istio-proxy"
Nov 03 20:41:03.295480 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.294497    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="a223486f-1753-4dd1-8fd5-a6c03d664555" containerName="istio-proxy"
Nov 03 20:41:03.295480 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.294583    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7880a3e1-aceb-4444-9615-bf168854a75b" containerName="revoke-keys"
Nov 03 20:41:03.295480 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.294599    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="73a74bc3-6a5f-46be-8094-74d859931e19" containerName="istio-proxy"
Nov 03 20:41:03.301021 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.300995    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.324310 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.324273    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"]
Nov 03 20:41:03.407112 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.407078    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.407327 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.407119    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.407327 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.407155    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rtlr\" (UniqueName: \"kubernetes.io/projected/49509a71-78ac-4412-85f6-7d51a7f0bc5d-kube-api-access-7rtlr\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.407327 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.407285    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.407327 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.407325    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.407483 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.407438    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.407483 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.407467    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.407549 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.407506    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.407629 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.407615    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.508867 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.508826    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.509050 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.508880    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.509050 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.508915    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rtlr\" (UniqueName: \"kubernetes.io/projected/49509a71-78ac-4412-85f6-7d51a7f0bc5d-kube-api-access-7rtlr\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.509050 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.508947    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.509050 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.508979    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.509050 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.509045    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.509332 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.509072    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.509332 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.509129    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.509332 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.509248    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.509518 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.509485    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-credential-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.509518 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.509510    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-data\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.509636 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.509596    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-workload-socket\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.509704 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.509664    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-workload-certs\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.509893 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.509867    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istiod-ca-cert\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.511459 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.511433    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-envoy\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.511592 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.511574    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-podinfo\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.521530 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.521506    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-token\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.521913 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.521892    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rtlr\" (UniqueName: \"kubernetes.io/projected/49509a71-78ac-4412-85f6-7d51a7f0bc5d-kube-api-access-7rtlr\") pod \"e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") " pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.619577 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.619426    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:03.910797 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.910770    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"]
Nov 03 20:41:03.910907 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:41:03.910841    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49509a71_78ac_4412_85f6_7d51a7f0bc5d.slice/crio-552eff421e8016c2acd2f9519c60eee25f2600ad0813902a44eaf2ed28137aef WatchSource:0}: Error finding container 552eff421e8016c2acd2f9519c60eee25f2600ad0813902a44eaf2ed28137aef: Status 404 returned error can't find the container with id 552eff421e8016c2acd2f9519c60eee25f2600ad0813902a44eaf2ed28137aef
Nov 03 20:41:03.913060 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.913028    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:41:03.913138 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.913094    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:41:03.913138 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:03.913126    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:41:04.206799 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:04.206707    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc" event={"ID":"49509a71-78ac-4412-85f6-7d51a7f0bc5d","Type":"ContainerStarted","Data":"356b71c7f075be707a3f129e088e23e508c2b3f1827f8e76d3bcea95175ae78b"}
Nov 03 20:41:04.206799 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:04.206781    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc" event={"ID":"49509a71-78ac-4412-85f6-7d51a7f0bc5d","Type":"ContainerStarted","Data":"552eff421e8016c2acd2f9519c60eee25f2600ad0813902a44eaf2ed28137aef"}
Nov 03 20:41:04.231006 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:04.230951    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc" podStartSLOduration=1.230937167 podStartE2EDuration="1.230937167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:04.229162899 +0000 UTC m=+3813.001455917" watchObservedRunningTime="2026-11-03 20:41:04.230937167 +0000 UTC m=+3813.003230185"
Nov 03 20:41:04.620503 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:04.620471    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:04.622124 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:04.622094    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" start-of-body=
Nov 03 20:41:04.622292 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:04.622149    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc" podUID="49509a71-78ac-4412-85f6-7d51a7f0bc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused"
Nov 03 20:41:05.620664 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:05.620626    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" start-of-body=
Nov 03 20:41:05.621175 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:05.620680    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc" podUID="49509a71-78ac-4412-85f6-7d51a7f0bc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused"
Nov 03 20:41:06.620009 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:06.619975    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" start-of-body=
Nov 03 20:41:06.620234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:06.620031    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc" podUID="49509a71-78ac-4412-85f6-7d51a7f0bc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused"
Nov 03 20:41:07.619837 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:07.619804    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" start-of-body=
Nov 03 20:41:07.620315 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:07.619858    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc" podUID="49509a71-78ac-4412-85f6-7d51a7f0bc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused"
Nov 03 20:41:08.620045 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:08.620008    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" start-of-body=
Nov 03 20:41:08.620492 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:08.620063    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc" podUID="49509a71-78ac-4412-85f6-7d51a7f0bc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused"
Nov 03 20:41:09.620037 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:09.620001    5591 patch_prober.go:29] interesting pod/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused" start-of-body=
Nov 03 20:41:09.620240 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:09.620059    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc" podUID="49509a71-78ac-4412-85f6-7d51a7f0bc5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.24:15021/healthz/ready\": dial tcp 10.134.1.24:15021: connect: connection refused"
Nov 03 20:41:10.624469 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:10.624438    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:10.624961 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:10.624901    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:10.625519 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:10.625502    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:11.277469 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:11.277430    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"]
Nov 03 20:41:11.278960 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:11.278916    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r" podUID="8e00f1a1-8156-4fa7-8308-b48127f82aa6" containerName="istio-proxy" containerID="cri-o://6b6268803ecbc18a4e4ac07fc79f3ad4034cc8ff312a3cdefe50da49f8e83cd0" gracePeriod=30
Nov 03 20:41:11.286152 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:11.286121    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"]
Nov 03 20:41:12.252313 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:12.252272    5591 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc" secret="" err="secret \"e2e-worker-w3-9f46e5-openshift-default-dockercfg-4djl8\" not found"
Nov 03 20:41:13.257774 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:13.257720    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc" podUID="49509a71-78ac-4412-85f6-7d51a7f0bc5d" containerName="istio-proxy" containerID="cri-o://356b71c7f075be707a3f129e088e23e508c2b3f1827f8e76d3bcea95175ae78b" gracePeriod=30
Nov 03 20:41:13.475156 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:13.475127    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-fz8hm_ea095e27-20df-40bb-a4c4-3997bd5e9cff/limitador/0.log"
Nov 03 20:41:14.595120 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:14.595084    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-79b8b796f-2sshp_747740b4-92e0-4179-9318-04fd9030b8e0/manager/0.log"
Nov 03 20:41:14.623618 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:14.623580    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"]
Nov 03 20:41:14.625276 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:14.623867    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh" podUID="f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a" containerName="istio-proxy" containerID="cri-o://a196faa4cabf6c21f395ccab70ffa4b179287c34fa6708e3078d53bb9d6dee66" gracePeriod=30
Nov 03 20:41:14.812260 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:14.812221    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-vx6gv_67d4147a-49d0-43ad-983d-cc033e4fc21a/manager/0.log"
Nov 03 20:41:16.110453 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.110412    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-55c765799f-2bvrd_7ea4d469-fff5-4de8-825a-7a76de2c46b2/manager/0.log"
Nov 03 20:41:16.544502 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.544458    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 20:41:16.650683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.650585    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e00f1a1-8156-4fa7-8308-b48127f82aa6-kube-api-access-znbm4\" (UniqueName: \"kubernetes.io/projected/8e00f1a1-8156-4fa7-8308-b48127f82aa6-kube-api-access-znbm4\") pod \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") "
Nov 03 20:41:16.650683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.650635    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-credential-socket\") pod \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") "
Nov 03 20:41:16.650683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.650680    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-token\" (UniqueName: \"kubernetes.io/projected/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-token\") pod \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") "
Nov 03 20:41:16.650996 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.650727    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-podinfo\") pod \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") "
Nov 03 20:41:16.650996 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.650763    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-envoy\") pod \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") "
Nov 03 20:41:16.650996 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.650812    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istiod-ca-cert\") pod \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") "
Nov 03 20:41:16.650996 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.650884    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-workload-certs\") pod \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") "
Nov 03 20:41:16.650996 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.650922    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-data\") pod \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") "
Nov 03 20:41:16.651334 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.651010    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-credential-socket" pod "8e00f1a1-8156-4fa7-8308-b48127f82aa6" (UID: "8e00f1a1-8156-4fa7-8308-b48127f82aa6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:41:16.651334 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.651023    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-workload-socket\") pod \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\" (UID: \"8e00f1a1-8156-4fa7-8308-b48127f82aa6\") "
Nov 03 20:41:16.651334 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.651144    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-workload-certs" pod "8e00f1a1-8156-4fa7-8308-b48127f82aa6" (UID: "8e00f1a1-8156-4fa7-8308-b48127f82aa6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:41:16.651334 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.651274    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-workload-socket" pod "8e00f1a1-8156-4fa7-8308-b48127f82aa6" (UID: "8e00f1a1-8156-4fa7-8308-b48127f82aa6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:41:16.651334 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.651294    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istiod-ca-cert" pod "8e00f1a1-8156-4fa7-8308-b48127f82aa6" (UID: "8e00f1a1-8156-4fa7-8308-b48127f82aa6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:41:16.651563 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.651435    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-data" pod "8e00f1a1-8156-4fa7-8308-b48127f82aa6" (UID: "8e00f1a1-8156-4fa7-8308-b48127f82aa6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:41:16.651826 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.651802    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:16.651877 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.651834    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:16.651877 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.651852    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:16.651877 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.651866    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:16.651977 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.651879    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:16.653221 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.653175    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-token" pod "8e00f1a1-8156-4fa7-8308-b48127f82aa6" (UID: "8e00f1a1-8156-4fa7-8308-b48127f82aa6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:41:16.653414 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.653393    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-envoy" pod "8e00f1a1-8156-4fa7-8308-b48127f82aa6" (UID: "8e00f1a1-8156-4fa7-8308-b48127f82aa6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:41:16.653625 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.653606    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-podinfo" pod "8e00f1a1-8156-4fa7-8308-b48127f82aa6" (UID: "8e00f1a1-8156-4fa7-8308-b48127f82aa6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:41:16.653669 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.653616    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e00f1a1-8156-4fa7-8308-b48127f82aa6-kube-api-access-znbm4" pod "8e00f1a1-8156-4fa7-8308-b48127f82aa6" (UID: "8e00f1a1-8156-4fa7-8308-b48127f82aa6"). InnerVolumeSpecName "kube-api-access-znbm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:41:16.753130 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.753082    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znbm4\" (UniqueName: \"kubernetes.io/projected/8e00f1a1-8156-4fa7-8308-b48127f82aa6-kube-api-access-znbm4\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:16.753130 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.753120    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:16.753130 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.753133    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:16.753423 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:16.753146    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e00f1a1-8156-4fa7-8308-b48127f82aa6-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:17.284645 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:17.284606    5591 generic.go:356] "Generic (PLEG): container finished" podID="8e00f1a1-8156-4fa7-8308-b48127f82aa6" containerID="6b6268803ecbc18a4e4ac07fc79f3ad4034cc8ff312a3cdefe50da49f8e83cd0" exitCode=0
Nov 03 20:41:17.285076 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:17.284667    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"
Nov 03 20:41:17.285076 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:17.284670    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r" event={"ID":"8e00f1a1-8156-4fa7-8308-b48127f82aa6","Type":"ContainerDied","Data":"6b6268803ecbc18a4e4ac07fc79f3ad4034cc8ff312a3cdefe50da49f8e83cd0"}
Nov 03 20:41:17.285076 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:17.284820    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r" event={"ID":"8e00f1a1-8156-4fa7-8308-b48127f82aa6","Type":"ContainerDied","Data":"c80c5955640014d9c6083295d31e98784534ca733a4f90f8331c64acf63db028"}
Nov 03 20:41:17.285076 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:17.284849    5591 scope.go:122] "RemoveContainer" containerID="6b6268803ecbc18a4e4ac07fc79f3ad4034cc8ff312a3cdefe50da49f8e83cd0"
Nov 03 20:41:17.295513 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:17.295492    5591 scope.go:122] "RemoveContainer" containerID="da60d0a13ec638f966a0e24803738a66382b122c2968c8495a32494fe926366e"
Nov 03 20:41:17.304541 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:17.304521    5591 scope.go:122] "RemoveContainer" containerID="6b6268803ecbc18a4e4ac07fc79f3ad4034cc8ff312a3cdefe50da49f8e83cd0"
Nov 03 20:41:17.304839 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:41:17.304819    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b6268803ecbc18a4e4ac07fc79f3ad4034cc8ff312a3cdefe50da49f8e83cd0\": container with ID starting with 6b6268803ecbc18a4e4ac07fc79f3ad4034cc8ff312a3cdefe50da49f8e83cd0 not found: ID does not exist" containerID="6b6268803ecbc18a4e4ac07fc79f3ad4034cc8ff312a3cdefe50da49f8e83cd0"
Nov 03 20:41:17.304911 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:17.304852    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b6268803ecbc18a4e4ac07fc79f3ad4034cc8ff312a3cdefe50da49f8e83cd0"} err="failed to get container status \"6b6268803ecbc18a4e4ac07fc79f3ad4034cc8ff312a3cdefe50da49f8e83cd0\": rpc error: code = NotFound desc = could not find container \"6b6268803ecbc18a4e4ac07fc79f3ad4034cc8ff312a3cdefe50da49f8e83cd0\": container with ID starting with 6b6268803ecbc18a4e4ac07fc79f3ad4034cc8ff312a3cdefe50da49f8e83cd0 not found: ID does not exist"
Nov 03 20:41:17.304911 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:17.304869    5591 scope.go:122] "RemoveContainer" containerID="da60d0a13ec638f966a0e24803738a66382b122c2968c8495a32494fe926366e"
Nov 03 20:41:17.305111 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:41:17.305095    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da60d0a13ec638f966a0e24803738a66382b122c2968c8495a32494fe926366e\": container with ID starting with da60d0a13ec638f966a0e24803738a66382b122c2968c8495a32494fe926366e not found: ID does not exist" containerID="da60d0a13ec638f966a0e24803738a66382b122c2968c8495a32494fe926366e"
Nov 03 20:41:17.305153 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:17.305116    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da60d0a13ec638f966a0e24803738a66382b122c2968c8495a32494fe926366e"} err="failed to get container status \"da60d0a13ec638f966a0e24803738a66382b122c2968c8495a32494fe926366e\": rpc error: code = NotFound desc = could not find container \"da60d0a13ec638f966a0e24803738a66382b122c2968c8495a32494fe926366e\": container with ID starting with da60d0a13ec638f966a0e24803738a66382b122c2968c8495a32494fe926366e not found: ID does not exist"
Nov 03 20:41:17.312542 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:17.312517    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"]
Nov 03 20:41:17.315091 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:17.315059    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-66m2r"]
Nov 03 20:41:17.347936 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:17.347907    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e00f1a1-8156-4fa7-8308-b48127f82aa6" path="/var/lib/kubelet/pods/8e00f1a1-8156-4fa7-8308-b48127f82aa6/volumes"
Nov 03 20:41:18.513509 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.513485    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:18.673708 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.673619    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-workload-socket\") pod \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") "
Nov 03 20:41:18.673708 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.673685    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-data\") pod \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") "
Nov 03 20:41:18.673942 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.673730    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-envoy\") pod \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") "
Nov 03 20:41:18.673942 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.673761    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-podinfo\") pod \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") "
Nov 03 20:41:18.673942 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.673809    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-credential-socket\") pod \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") "
Nov 03 20:41:18.673942 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.673857    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49509a71-78ac-4412-85f6-7d51a7f0bc5d-kube-api-access-7rtlr\" (UniqueName: \"kubernetes.io/projected/49509a71-78ac-4412-85f6-7d51a7f0bc5d-kube-api-access-7rtlr\") pod \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") "
Nov 03 20:41:18.673942 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.673935    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istiod-ca-cert\") pod \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") "
Nov 03 20:41:18.674245 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.673972    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-token\" (UniqueName: \"kubernetes.io/projected/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-token\") pod \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") "
Nov 03 20:41:18.674245 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.673966    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-workload-socket" pod "49509a71-78ac-4412-85f6-7d51a7f0bc5d" (UID: "49509a71-78ac-4412-85f6-7d51a7f0bc5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:41:18.674245 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.674050    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-workload-certs\") pod \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\" (UID: \"49509a71-78ac-4412-85f6-7d51a7f0bc5d\") "
Nov 03 20:41:18.674440 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.674255    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-credential-socket" pod "49509a71-78ac-4412-85f6-7d51a7f0bc5d" (UID: "49509a71-78ac-4412-85f6-7d51a7f0bc5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:41:18.674440 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.674354    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istiod-ca-cert" pod "49509a71-78ac-4412-85f6-7d51a7f0bc5d" (UID: "49509a71-78ac-4412-85f6-7d51a7f0bc5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:41:18.674566 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.674454    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-workload-certs" pod "49509a71-78ac-4412-85f6-7d51a7f0bc5d" (UID: "49509a71-78ac-4412-85f6-7d51a7f0bc5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:41:18.674566 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.674554    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-data" pod "49509a71-78ac-4412-85f6-7d51a7f0bc5d" (UID: "49509a71-78ac-4412-85f6-7d51a7f0bc5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:41:18.674686 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.674568    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:18.674686 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.674590    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:18.674686 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.674607    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:18.674686 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.674651    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:18.676275 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.676251    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49509a71-78ac-4412-85f6-7d51a7f0bc5d-kube-api-access-7rtlr" pod "49509a71-78ac-4412-85f6-7d51a7f0bc5d" (UID: "49509a71-78ac-4412-85f6-7d51a7f0bc5d"). InnerVolumeSpecName "kube-api-access-7rtlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:41:18.676368 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.676329    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-token" pod "49509a71-78ac-4412-85f6-7d51a7f0bc5d" (UID: "49509a71-78ac-4412-85f6-7d51a7f0bc5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:41:18.676773 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.676747    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-envoy" pod "49509a71-78ac-4412-85f6-7d51a7f0bc5d" (UID: "49509a71-78ac-4412-85f6-7d51a7f0bc5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:41:18.676773 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.676755    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-podinfo" pod "49509a71-78ac-4412-85f6-7d51a7f0bc5d" (UID: "49509a71-78ac-4412-85f6-7d51a7f0bc5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:41:18.776055 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.776006    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:18.776055 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.776049    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:18.776055 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.776059    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:18.776055 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.776069    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rtlr\" (UniqueName: \"kubernetes.io/projected/49509a71-78ac-4412-85f6-7d51a7f0bc5d-kube-api-access-7rtlr\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:18.776377 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:18.776079    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49509a71-78ac-4412-85f6-7d51a7f0bc5d-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:19.296607 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.296568    5591 generic.go:356] "Generic (PLEG): container finished" podID="49509a71-78ac-4412-85f6-7d51a7f0bc5d" containerID="356b71c7f075be707a3f129e088e23e508c2b3f1827f8e76d3bcea95175ae78b" exitCode=0
Nov 03 20:41:19.296864 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.296611    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"
Nov 03 20:41:19.296864 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.296651    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc" event={"ID":"49509a71-78ac-4412-85f6-7d51a7f0bc5d","Type":"ContainerDied","Data":"356b71c7f075be707a3f129e088e23e508c2b3f1827f8e76d3bcea95175ae78b"}
Nov 03 20:41:19.296864 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.296708    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc" event={"ID":"49509a71-78ac-4412-85f6-7d51a7f0bc5d","Type":"ContainerDied","Data":"552eff421e8016c2acd2f9519c60eee25f2600ad0813902a44eaf2ed28137aef"}
Nov 03 20:41:19.296864 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.296739    5591 scope.go:122] "RemoveContainer" containerID="356b71c7f075be707a3f129e088e23e508c2b3f1827f8e76d3bcea95175ae78b"
Nov 03 20:41:19.306846 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.306828    5591 scope.go:122] "RemoveContainer" containerID="356b71c7f075be707a3f129e088e23e508c2b3f1827f8e76d3bcea95175ae78b"
Nov 03 20:41:19.307131 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:41:19.307112    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"356b71c7f075be707a3f129e088e23e508c2b3f1827f8e76d3bcea95175ae78b\": container with ID starting with 356b71c7f075be707a3f129e088e23e508c2b3f1827f8e76d3bcea95175ae78b not found: ID does not exist" containerID="356b71c7f075be707a3f129e088e23e508c2b3f1827f8e76d3bcea95175ae78b"
Nov 03 20:41:19.307194 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.307139    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"356b71c7f075be707a3f129e088e23e508c2b3f1827f8e76d3bcea95175ae78b"} err="failed to get container status \"356b71c7f075be707a3f129e088e23e508c2b3f1827f8e76d3bcea95175ae78b\": rpc error: code = NotFound desc = could not find container \"356b71c7f075be707a3f129e088e23e508c2b3f1827f8e76d3bcea95175ae78b\": container with ID starting with 356b71c7f075be707a3f129e088e23e508c2b3f1827f8e76d3bcea95175ae78b not found: ID does not exist"
Nov 03 20:41:19.321166 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.321133    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"]
Nov 03 20:41:19.325938 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.325911    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9f46e5-openshift-default-55dbb6f54f-sgtzc"]
Nov 03 20:41:19.347920 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.347893    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49509a71-78ac-4412-85f6-7d51a7f0bc5d" path="/var/lib/kubelet/pods/49509a71-78ac-4412-85f6-7d51a7f0bc5d/volumes"
Nov 03 20:41:19.883407 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.883382    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:41:19.989435 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.989359    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-kube-api-access-p2nn9\" (UniqueName: \"kubernetes.io/projected/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-kube-api-access-p2nn9\") pod \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") "
Nov 03 20:41:19.989435 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.989395    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istiod-ca-cert\") pod \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") "
Nov 03 20:41:19.989435 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.989426    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-workload-certs\") pod \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") "
Nov 03 20:41:19.989710 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.989450    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-data\") pod \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") "
Nov 03 20:41:19.989710 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.989517    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-podinfo\") pod \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") "
Nov 03 20:41:19.989710 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.989556    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-credential-socket\") pod \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") "
Nov 03 20:41:19.989710 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.989595    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-workload-socket\") pod \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") "
Nov 03 20:41:19.989710 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.989673    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-token\" (UniqueName: \"kubernetes.io/projected/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-token\") pod \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") "
Nov 03 20:41:19.989999 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.989706    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-workload-certs" pod "f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a" (UID: "f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:41:19.989999 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.989759    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-envoy\") pod \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\" (UID: \"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a\") "
Nov 03 20:41:19.989999 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.989897    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-data" pod "f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a" (UID: "f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:41:19.989999 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.989940    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-credential-socket" pod "f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a" (UID: "f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:41:19.989999 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.989951    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-workload-socket" pod "f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a" (UID: "f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:41:19.989999 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.989974    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istiod-ca-cert" pod "f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a" (UID: "f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:41:19.990335 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.990177    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:19.990335 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.990220    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:19.990335 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.990238    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:19.990335 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.990253    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:19.990335 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.990267    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:19.992018 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.991986    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-kube-api-access-p2nn9" pod "f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a" (UID: "f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a"). InnerVolumeSpecName "kube-api-access-p2nn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:41:19.992018 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.992009    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-token" pod "f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a" (UID: "f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:41:19.992258 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.992022    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-podinfo" pod "f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a" (UID: "f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:41:19.992258 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:19.992034    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-envoy" pod "f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a" (UID: "f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:41:20.091528 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:20.091482    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:20.091528 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:20.091517    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:20.091528 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:20.091531    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:20.091782 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:20.091543    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2nn9\" (UniqueName: \"kubernetes.io/projected/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a-kube-api-access-p2nn9\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:20.304425 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:20.304384    5591 generic.go:356] "Generic (PLEG): container finished" podID="f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a" containerID="a196faa4cabf6c21f395ccab70ffa4b179287c34fa6708e3078d53bb9d6dee66" exitCode=0
Nov 03 20:41:20.304631 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:20.304429    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh" event={"ID":"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a","Type":"ContainerDied","Data":"a196faa4cabf6c21f395ccab70ffa4b179287c34fa6708e3078d53bb9d6dee66"}
Nov 03 20:41:20.304631 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:20.304482    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh" event={"ID":"f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a","Type":"ContainerDied","Data":"9c8456afe0fa29e401f40b15d41180be52bb3dff42e88161e5f0c251a0f044ab"}
Nov 03 20:41:20.304631 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:20.304505    5591 scope.go:122] "RemoveContainer" containerID="a196faa4cabf6c21f395ccab70ffa4b179287c34fa6708e3078d53bb9d6dee66"
Nov 03 20:41:20.304631 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:20.304437    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"
Nov 03 20:41:20.314630 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:20.314612    5591 scope.go:122] "RemoveContainer" containerID="a196faa4cabf6c21f395ccab70ffa4b179287c34fa6708e3078d53bb9d6dee66"
Nov 03 20:41:20.314931 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:41:20.314910    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a196faa4cabf6c21f395ccab70ffa4b179287c34fa6708e3078d53bb9d6dee66\": container with ID starting with a196faa4cabf6c21f395ccab70ffa4b179287c34fa6708e3078d53bb9d6dee66 not found: ID does not exist" containerID="a196faa4cabf6c21f395ccab70ffa4b179287c34fa6708e3078d53bb9d6dee66"
Nov 03 20:41:20.314979 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:20.314941    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a196faa4cabf6c21f395ccab70ffa4b179287c34fa6708e3078d53bb9d6dee66"} err="failed to get container status \"a196faa4cabf6c21f395ccab70ffa4b179287c34fa6708e3078d53bb9d6dee66\": rpc error: code = NotFound desc = could not find container \"a196faa4cabf6c21f395ccab70ffa4b179287c34fa6708e3078d53bb9d6dee66\": container with ID starting with a196faa4cabf6c21f395ccab70ffa4b179287c34fa6708e3078d53bb9d6dee66 not found: ID does not exist"
Nov 03 20:41:20.327178 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:20.327150    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"]
Nov 03 20:41:20.331238 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:20.331194    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-tsrbh"]
Nov 03 20:41:21.355337 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:21.355292    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a" path="/var/lib/kubelet/pods/f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a/volumes"
Nov 03 20:41:28.723393 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:41:28.723342    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:41:37.954171 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:37.954136    5591 scope.go:122] "RemoveContainer" containerID="0f19d6863d124322e0a6ffba90bd2d002acc852418bdd319f483f6a8b900a5f4"
Nov 03 20:41:47.348262 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:47.348230    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:41:55.037329 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.037288    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d790d69a-8429c"]
Nov 03 20:41:55.037927 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.037904    5591 cpu_manager.go:414] "containerMap: removing container" podUID="f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a" containerName="istio-proxy"
Nov 03 20:41:55.038047 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.037930    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a" containerName="istio-proxy"
Nov 03 20:41:55.038047 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.037950    5591 cpu_manager.go:414] "containerMap: removing container" podUID="49509a71-78ac-4412-85f6-7d51a7f0bc5d" containerName="istio-proxy"
Nov 03 20:41:55.038047 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.037958    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49509a71-78ac-4412-85f6-7d51a7f0bc5d" containerName="istio-proxy"
Nov 03 20:41:55.038047 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.037975    5591 cpu_manager.go:414] "containerMap: removing container" podUID="8e00f1a1-8156-4fa7-8308-b48127f82aa6" containerName="istio-proxy"
Nov 03 20:41:55.038047 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.037983    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e00f1a1-8156-4fa7-8308-b48127f82aa6" containerName="istio-proxy"
Nov 03 20:41:55.038405 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.038118    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="49509a71-78ac-4412-85f6-7d51a7f0bc5d" containerName="istio-proxy"
Nov 03 20:41:55.038405 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.038135    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e00f1a1-8156-4fa7-8308-b48127f82aa6" containerName="istio-proxy"
Nov 03 20:41:55.038405 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.038156    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f5c7f2-56a6-4f8e-ade2-7da70f8ebb7a" containerName="istio-proxy"
Nov 03 20:41:55.043464 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.043442    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d790d69a-8429c"
Nov 03 20:41:55.048233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.048163    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:41:55.048956 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.048927    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:41:55.049091 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.048950    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:41:55.049162 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.049102    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ttqqd\""
Nov 03 20:41:55.053382 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.051162    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d790d69a-8429c"]
Nov 03 20:41:55.137843 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.137810    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq9k5\" (UniqueName: \"kubernetes.io/projected/a67f63d6-ab9e-4117-b9b3-1b8928015058-kube-api-access-xq9k5\") pod \"maas-api-revoke-keys-e2e-ait-d790d69a-8429c\" (UID: \"a67f63d6-ab9e-4117-b9b3-1b8928015058\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d790d69a-8429c"
Nov 03 20:41:55.138022 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.137945    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67f63d6-ab9e-4117-b9b3-1b8928015058-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d790d69a-8429c\" (UID: \"a67f63d6-ab9e-4117-b9b3-1b8928015058\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d790d69a-8429c"
Nov 03 20:41:55.138067 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.138019    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67f63d6-ab9e-4117-b9b3-1b8928015058-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d790d69a-8429c\" (UID: \"a67f63d6-ab9e-4117-b9b3-1b8928015058\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d790d69a-8429c"
Nov 03 20:41:55.239192 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.239144    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67f63d6-ab9e-4117-b9b3-1b8928015058-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d790d69a-8429c\" (UID: \"a67f63d6-ab9e-4117-b9b3-1b8928015058\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d790d69a-8429c"
Nov 03 20:41:55.239413 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.239275    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67f63d6-ab9e-4117-b9b3-1b8928015058-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d790d69a-8429c\" (UID: \"a67f63d6-ab9e-4117-b9b3-1b8928015058\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d790d69a-8429c"
Nov 03 20:41:55.239413 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.239308    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq9k5\" (UniqueName: \"kubernetes.io/projected/a67f63d6-ab9e-4117-b9b3-1b8928015058-kube-api-access-xq9k5\") pod \"maas-api-revoke-keys-e2e-ait-d790d69a-8429c\" (UID: \"a67f63d6-ab9e-4117-b9b3-1b8928015058\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d790d69a-8429c"
Nov 03 20:41:55.239894 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.239868    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67f63d6-ab9e-4117-b9b3-1b8928015058-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d790d69a-8429c\" (UID: \"a67f63d6-ab9e-4117-b9b3-1b8928015058\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d790d69a-8429c"
Nov 03 20:41:55.242002 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.241977    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67f63d6-ab9e-4117-b9b3-1b8928015058-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d790d69a-8429c\" (UID: \"a67f63d6-ab9e-4117-b9b3-1b8928015058\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d790d69a-8429c"
Nov 03 20:41:55.252539 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.252512    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq9k5\" (UniqueName: \"kubernetes.io/projected/a67f63d6-ab9e-4117-b9b3-1b8928015058-kube-api-access-xq9k5\") pod \"maas-api-revoke-keys-e2e-ait-d790d69a-8429c\" (UID: \"a67f63d6-ab9e-4117-b9b3-1b8928015058\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d790d69a-8429c"
Nov 03 20:41:55.358598 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.358520    5591 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d790d69a-8429c" secret="" err="secret \"maas-api-cleanup-dockercfg-ttqqd\" not found"
Nov 03 20:41:55.358598 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.358571    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d790d69a-8429c"
Nov 03 20:41:55.637867 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:55.637843    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d790d69a-8429c"]
Nov 03 20:41:55.640973 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:41:55.640946    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67f63d6_ab9e_4117_b9b3_1b8928015058.slice/crio-ecc2187bbb2da521f5cd0129c12ad2abd947e147b8cae5ccddd778f39feed1d8 WatchSource:0}: Error finding container ecc2187bbb2da521f5cd0129c12ad2abd947e147b8cae5ccddd778f39feed1d8: Status 404 returned error can't find the container with id ecc2187bbb2da521f5cd0129c12ad2abd947e147b8cae5ccddd778f39feed1d8
Nov 03 20:41:56.496741 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:56.496711    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d790d69a-8429c" event={"ID":"a67f63d6-ab9e-4117-b9b3-1b8928015058","Type":"ContainerStarted","Data":"ffb0c556089aee49a4605053b6976adab1144e2cdb93f2b183e0533619504763"}
Nov 03 20:41:56.497093 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:56.496775    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d790d69a-8429c" event={"ID":"a67f63d6-ab9e-4117-b9b3-1b8928015058","Type":"ContainerStarted","Data":"ecc2187bbb2da521f5cd0129c12ad2abd947e147b8cae5ccddd778f39feed1d8"}
Nov 03 20:41:57.503602 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:57.503569    5591 generic.go:356] "Generic (PLEG): container finished" podID="a67f63d6-ab9e-4117-b9b3-1b8928015058" containerID="ffb0c556089aee49a4605053b6976adab1144e2cdb93f2b183e0533619504763" exitCode=0
Nov 03 20:41:57.504106 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:57.503668    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d790d69a-8429c" event={"ID":"a67f63d6-ab9e-4117-b9b3-1b8928015058","Type":"ContainerDied","Data":"ffb0c556089aee49a4605053b6976adab1144e2cdb93f2b183e0533619504763"}
Nov 03 20:41:57.640980 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:57.640954    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d790d69a-8429c"
Nov 03 20:41:57.765581 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:57.765553    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a67f63d6-ab9e-4117-b9b3-1b8928015058-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67f63d6-ab9e-4117-b9b3-1b8928015058-maas-api-openshift-ca\") pod \"a67f63d6-ab9e-4117-b9b3-1b8928015058\" (UID: \"a67f63d6-ab9e-4117-b9b3-1b8928015058\") "
Nov 03 20:41:57.765791 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:57.765717    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a67f63d6-ab9e-4117-b9b3-1b8928015058-kube-api-access-xq9k5\" (UniqueName: \"kubernetes.io/projected/a67f63d6-ab9e-4117-b9b3-1b8928015058-kube-api-access-xq9k5\") pod \"a67f63d6-ab9e-4117-b9b3-1b8928015058\" (UID: \"a67f63d6-ab9e-4117-b9b3-1b8928015058\") "
Nov 03 20:41:57.765861 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:57.765806    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a67f63d6-ab9e-4117-b9b3-1b8928015058-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67f63d6-ab9e-4117-b9b3-1b8928015058-maas-api-xks-ca\") pod \"a67f63d6-ab9e-4117-b9b3-1b8928015058\" (UID: \"a67f63d6-ab9e-4117-b9b3-1b8928015058\") "
Nov 03 20:41:57.765927 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:57.765904    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a67f63d6-ab9e-4117-b9b3-1b8928015058-maas-api-openshift-ca" pod "a67f63d6-ab9e-4117-b9b3-1b8928015058" (UID: "a67f63d6-ab9e-4117-b9b3-1b8928015058"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:41:57.766137 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:57.766107    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a67f63d6-ab9e-4117-b9b3-1b8928015058-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:57.768111 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:57.768087    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a67f63d6-ab9e-4117-b9b3-1b8928015058-maas-api-xks-ca" pod "a67f63d6-ab9e-4117-b9b3-1b8928015058" (UID: "a67f63d6-ab9e-4117-b9b3-1b8928015058"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:41:57.768111 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:57.768098    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a67f63d6-ab9e-4117-b9b3-1b8928015058-kube-api-access-xq9k5" pod "a67f63d6-ab9e-4117-b9b3-1b8928015058" (UID: "a67f63d6-ab9e-4117-b9b3-1b8928015058"). InnerVolumeSpecName "kube-api-access-xq9k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:41:57.867195 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:57.867164    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq9k5\" (UniqueName: \"kubernetes.io/projected/a67f63d6-ab9e-4117-b9b3-1b8928015058-kube-api-access-xq9k5\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:57.867195 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:57.867190    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a67f63d6-ab9e-4117-b9b3-1b8928015058-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:41:58.510317 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:58.510273    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d790d69a-8429c"
Nov 03 20:41:58.510317 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:58.510252    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d790d69a-8429c" event={"ID":"a67f63d6-ab9e-4117-b9b3-1b8928015058","Type":"ContainerDied","Data":"ecc2187bbb2da521f5cd0129c12ad2abd947e147b8cae5ccddd778f39feed1d8"}
Nov 03 20:41:58.510850 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:41:58.510352    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecc2187bbb2da521f5cd0129c12ad2abd947e147b8cae5ccddd778f39feed1d8"
Nov 03 20:41:58.723289 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:41:58.723246    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:42:06.123521 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:06.123481    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d790d69a-8429c"]
Nov 03 20:42:06.128107 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:06.128080    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d790d69a-8429c"]
Nov 03 20:42:07.348446 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:07.348411    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a67f63d6-ab9e-4117-b9b3-1b8928015058" path="/var/lib/kubelet/pods/a67f63d6-ab9e-4117-b9b3-1b8928015058/volumes"
Nov 03 20:42:07.527809 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:07.527771    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:42:13.097131 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:13.097051    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55c765799f-2bvrd"]
Nov 03 20:42:13.097536 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:13.097300    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-55c765799f-2bvrd" podUID="7ea4d469-fff5-4de8-825a-7a76de2c46b2" containerName="manager" containerID="cri-o://4c6776abaeae656e3ead38c258b97bf6f12c41e559bedbddabe94a7865181005" gracePeriod=30
Nov 03 20:42:13.350504 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:13.350447    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55c765799f-2bvrd"
Nov 03 20:42:13.424448 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:13.424418    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ea4d469-fff5-4de8-825a-7a76de2c46b2-kube-api-access-scmm2\" (UniqueName: \"kubernetes.io/projected/7ea4d469-fff5-4de8-825a-7a76de2c46b2-kube-api-access-scmm2\") pod \"7ea4d469-fff5-4de8-825a-7a76de2c46b2\" (UID: \"7ea4d469-fff5-4de8-825a-7a76de2c46b2\") "
Nov 03 20:42:13.424636 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:13.424488    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ea4d469-fff5-4de8-825a-7a76de2c46b2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ea4d469-fff5-4de8-825a-7a76de2c46b2-metrics-certs\") pod \"7ea4d469-fff5-4de8-825a-7a76de2c46b2\" (UID: \"7ea4d469-fff5-4de8-825a-7a76de2c46b2\") "
Nov 03 20:42:13.424636 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:13.424520    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ea4d469-fff5-4de8-825a-7a76de2c46b2-webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ea4d469-fff5-4de8-825a-7a76de2c46b2-webhook-cert\") pod \"7ea4d469-fff5-4de8-825a-7a76de2c46b2\" (UID: \"7ea4d469-fff5-4de8-825a-7a76de2c46b2\") "
Nov 03 20:42:13.426758 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:13.426723    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea4d469-fff5-4de8-825a-7a76de2c46b2-kube-api-access-scmm2" pod "7ea4d469-fff5-4de8-825a-7a76de2c46b2" (UID: "7ea4d469-fff5-4de8-825a-7a76de2c46b2"). InnerVolumeSpecName "kube-api-access-scmm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:42:13.426887 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:13.426765    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ea4d469-fff5-4de8-825a-7a76de2c46b2-metrics-certs" pod "7ea4d469-fff5-4de8-825a-7a76de2c46b2" (UID: "7ea4d469-fff5-4de8-825a-7a76de2c46b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:42:13.427194 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:13.427168    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ea4d469-fff5-4de8-825a-7a76de2c46b2-webhook-cert" pod "7ea4d469-fff5-4de8-825a-7a76de2c46b2" (UID: "7ea4d469-fff5-4de8-825a-7a76de2c46b2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:42:13.526348 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:13.526315    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ea4d469-fff5-4de8-825a-7a76de2c46b2-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:42:13.526348 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:13.526345    5591 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ea4d469-fff5-4de8-825a-7a76de2c46b2-webhook-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:42:13.526578 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:13.526359    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scmm2\" (UniqueName: \"kubernetes.io/projected/7ea4d469-fff5-4de8-825a-7a76de2c46b2-kube-api-access-scmm2\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:42:13.587027 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:13.586991    5591 generic.go:356] "Generic (PLEG): container finished" podID="7ea4d469-fff5-4de8-825a-7a76de2c46b2" containerID="4c6776abaeae656e3ead38c258b97bf6f12c41e559bedbddabe94a7865181005" exitCode=0
Nov 03 20:42:13.587252 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:13.587051    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55c765799f-2bvrd"
Nov 03 20:42:13.587252 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:13.587079    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55c765799f-2bvrd" event={"ID":"7ea4d469-fff5-4de8-825a-7a76de2c46b2","Type":"ContainerDied","Data":"4c6776abaeae656e3ead38c258b97bf6f12c41e559bedbddabe94a7865181005"}
Nov 03 20:42:13.587252 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:13.587141    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55c765799f-2bvrd" event={"ID":"7ea4d469-fff5-4de8-825a-7a76de2c46b2","Type":"ContainerDied","Data":"1c3ac8573fc4d0a7e1c26df437725db746fffad0e8174883360074ad16a7efaf"}
Nov 03 20:42:13.587252 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:13.587175    5591 scope.go:122] "RemoveContainer" containerID="4c6776abaeae656e3ead38c258b97bf6f12c41e559bedbddabe94a7865181005"
Nov 03 20:42:13.599393 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:13.599373    5591 scope.go:122] "RemoveContainer" containerID="4c6776abaeae656e3ead38c258b97bf6f12c41e559bedbddabe94a7865181005"
Nov 03 20:42:13.599682 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:42:13.599654    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c6776abaeae656e3ead38c258b97bf6f12c41e559bedbddabe94a7865181005\": container with ID starting with 4c6776abaeae656e3ead38c258b97bf6f12c41e559bedbddabe94a7865181005 not found: ID does not exist" containerID="4c6776abaeae656e3ead38c258b97bf6f12c41e559bedbddabe94a7865181005"
Nov 03 20:42:13.599761 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:13.599695    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c6776abaeae656e3ead38c258b97bf6f12c41e559bedbddabe94a7865181005"} err="failed to get container status \"4c6776abaeae656e3ead38c258b97bf6f12c41e559bedbddabe94a7865181005\": rpc error: code = NotFound desc = could not find container \"4c6776abaeae656e3ead38c258b97bf6f12c41e559bedbddabe94a7865181005\": container with ID starting with 4c6776abaeae656e3ead38c258b97bf6f12c41e559bedbddabe94a7865181005 not found: ID does not exist"
Nov 03 20:42:13.609839 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:13.609786    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55c765799f-2bvrd"]
Nov 03 20:42:13.613721 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:13.613702    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-55c765799f-2bvrd"]
Nov 03 20:42:15.348442 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:15.348407    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ea4d469-fff5-4de8-825a-7a76de2c46b2" path="/var/lib/kubelet/pods/7ea4d469-fff5-4de8-825a-7a76de2c46b2/volumes"
Nov 03 20:42:28.722520 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:42:28.722472    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:42:34.148956 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:34.148923    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 20:42:34.151629 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:34.151610    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 20:42:44.771585 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:44.771546    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"]
Nov 03 20:42:44.772037 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:44.771999    5591 cpu_manager.go:414] "containerMap: removing container" podUID="8e00f1a1-8156-4fa7-8308-b48127f82aa6" containerName="istio-proxy"
Nov 03 20:42:44.772037 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:44.772011    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e00f1a1-8156-4fa7-8308-b48127f82aa6" containerName="istio-proxy"
Nov 03 20:42:44.772119 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:44.772051    5591 cpu_manager.go:414] "containerMap: removing container" podUID="7ea4d469-fff5-4de8-825a-7a76de2c46b2" containerName="manager"
Nov 03 20:42:44.772119 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:44.772059    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea4d469-fff5-4de8-825a-7a76de2c46b2" containerName="manager"
Nov 03 20:42:44.772119 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:44.772067    5591 cpu_manager.go:414] "containerMap: removing container" podUID="a67f63d6-ab9e-4117-b9b3-1b8928015058" containerName="revoke-keys"
Nov 03 20:42:44.772119 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:44.772072    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a67f63d6-ab9e-4117-b9b3-1b8928015058" containerName="revoke-keys"
Nov 03 20:42:44.772292 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:44.772134    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ea4d469-fff5-4de8-825a-7a76de2c46b2" containerName="manager"
Nov 03 20:42:44.772292 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:44.772144    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="a67f63d6-ab9e-4117-b9b3-1b8928015058" containerName="revoke-keys"
Nov 03 20:42:44.772292 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:44.772153    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e00f1a1-8156-4fa7-8308-b48127f82aa6" containerName="istio-proxy"
Nov 03 20:42:44.776499 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:44.776480    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:44.789905 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:44.789878    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"]
Nov 03 20:42:44.941025 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:44.940986    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/128a2ead-106d-483e-a4e0-8e13c7c91e28-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-x4bds\" (UID: \"128a2ead-106d-483e-a4e0-8e13c7c91e28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:44.941025 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:44.941038    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/128a2ead-106d-483e-a4e0-8e13c7c91e28-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-x4bds\" (UID: \"128a2ead-106d-483e-a4e0-8e13c7c91e28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:44.941299 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:44.941113    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/128a2ead-106d-483e-a4e0-8e13c7c91e28-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-x4bds\" (UID: \"128a2ead-106d-483e-a4e0-8e13c7c91e28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:44.941299 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:44.941171    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/128a2ead-106d-483e-a4e0-8e13c7c91e28-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-x4bds\" (UID: \"128a2ead-106d-483e-a4e0-8e13c7c91e28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:44.941299 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:44.941253    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/128a2ead-106d-483e-a4e0-8e13c7c91e28-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-x4bds\" (UID: \"128a2ead-106d-483e-a4e0-8e13c7c91e28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:44.941299 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:44.941293    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/128a2ead-106d-483e-a4e0-8e13c7c91e28-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-x4bds\" (UID: \"128a2ead-106d-483e-a4e0-8e13c7c91e28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:44.941437 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:44.941318    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vhdf\" (UniqueName: \"kubernetes.io/projected/128a2ead-106d-483e-a4e0-8e13c7c91e28-kube-api-access-5vhdf\") pod \"maas-default-gateway-openshift-default-855f9574cf-x4bds\" (UID: \"128a2ead-106d-483e-a4e0-8e13c7c91e28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:44.941437 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:44.941386    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/128a2ead-106d-483e-a4e0-8e13c7c91e28-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-x4bds\" (UID: \"128a2ead-106d-483e-a4e0-8e13c7c91e28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:44.941437 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:44.941406    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/128a2ead-106d-483e-a4e0-8e13c7c91e28-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-x4bds\" (UID: \"128a2ead-106d-483e-a4e0-8e13c7c91e28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:45.042612 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:45.042514    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/128a2ead-106d-483e-a4e0-8e13c7c91e28-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-x4bds\" (UID: \"128a2ead-106d-483e-a4e0-8e13c7c91e28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:45.042612 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:45.042555    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/128a2ead-106d-483e-a4e0-8e13c7c91e28-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-x4bds\" (UID: \"128a2ead-106d-483e-a4e0-8e13c7c91e28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:45.042612 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:45.042607    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/128a2ead-106d-483e-a4e0-8e13c7c91e28-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-x4bds\" (UID: \"128a2ead-106d-483e-a4e0-8e13c7c91e28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:45.042916 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:45.042636    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/128a2ead-106d-483e-a4e0-8e13c7c91e28-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-x4bds\" (UID: \"128a2ead-106d-483e-a4e0-8e13c7c91e28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:45.042916 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:45.042798    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/128a2ead-106d-483e-a4e0-8e13c7c91e28-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-x4bds\" (UID: \"128a2ead-106d-483e-a4e0-8e13c7c91e28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:45.042916 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:45.042859    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/128a2ead-106d-483e-a4e0-8e13c7c91e28-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-x4bds\" (UID: \"128a2ead-106d-483e-a4e0-8e13c7c91e28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:45.043087 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:45.042919    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/128a2ead-106d-483e-a4e0-8e13c7c91e28-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-x4bds\" (UID: \"128a2ead-106d-483e-a4e0-8e13c7c91e28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:45.043087 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:45.042926    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/128a2ead-106d-483e-a4e0-8e13c7c91e28-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-x4bds\" (UID: \"128a2ead-106d-483e-a4e0-8e13c7c91e28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:45.043087 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:45.042977    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/128a2ead-106d-483e-a4e0-8e13c7c91e28-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-x4bds\" (UID: \"128a2ead-106d-483e-a4e0-8e13c7c91e28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:45.043087 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:45.042998    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/128a2ead-106d-483e-a4e0-8e13c7c91e28-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-x4bds\" (UID: \"128a2ead-106d-483e-a4e0-8e13c7c91e28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:45.043087 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:45.043011    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/128a2ead-106d-483e-a4e0-8e13c7c91e28-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-x4bds\" (UID: \"128a2ead-106d-483e-a4e0-8e13c7c91e28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:45.043087 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:45.043036    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vhdf\" (UniqueName: \"kubernetes.io/projected/128a2ead-106d-483e-a4e0-8e13c7c91e28-kube-api-access-5vhdf\") pod \"maas-default-gateway-openshift-default-855f9574cf-x4bds\" (UID: \"128a2ead-106d-483e-a4e0-8e13c7c91e28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:45.043468 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:45.043268    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/128a2ead-106d-483e-a4e0-8e13c7c91e28-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-x4bds\" (UID: \"128a2ead-106d-483e-a4e0-8e13c7c91e28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:45.043587 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:45.043564    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/128a2ead-106d-483e-a4e0-8e13c7c91e28-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-x4bds\" (UID: \"128a2ead-106d-483e-a4e0-8e13c7c91e28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:45.045396 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:45.045367    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/128a2ead-106d-483e-a4e0-8e13c7c91e28-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-x4bds\" (UID: \"128a2ead-106d-483e-a4e0-8e13c7c91e28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:45.045526 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:45.045510    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/128a2ead-106d-483e-a4e0-8e13c7c91e28-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-x4bds\" (UID: \"128a2ead-106d-483e-a4e0-8e13c7c91e28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:45.055773 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:45.055746    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/128a2ead-106d-483e-a4e0-8e13c7c91e28-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-x4bds\" (UID: \"128a2ead-106d-483e-a4e0-8e13c7c91e28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:45.055927 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:45.055910    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vhdf\" (UniqueName: \"kubernetes.io/projected/128a2ead-106d-483e-a4e0-8e13c7c91e28-kube-api-access-5vhdf\") pod \"maas-default-gateway-openshift-default-855f9574cf-x4bds\" (UID: \"128a2ead-106d-483e-a4e0-8e13c7c91e28\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:45.093323 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:45.093286    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:45.375946 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:45.375896    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"]
Nov 03 20:42:45.378490 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:42:45.378465    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128a2ead_106d_483e_a4e0_8e13c7c91e28.slice/crio-ffcf83d4e1db5b4ef15c598023976af43d8a4cddcfd539b3af1724714dab0c6c WatchSource:0}: Error finding container ffcf83d4e1db5b4ef15c598023976af43d8a4cddcfd539b3af1724714dab0c6c: Status 404 returned error can't find the container with id ffcf83d4e1db5b4ef15c598023976af43d8a4cddcfd539b3af1724714dab0c6c
Nov 03 20:42:45.380560 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:45.380526    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:42:45.380635 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:45.380595    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:42:45.380635 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:45.380623    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:42:45.745433 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:45.745334    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds" event={"ID":"128a2ead-106d-483e-a4e0-8e13c7c91e28","Type":"ContainerStarted","Data":"a29e02f80efd3c6492cd21e8d97e47ace1b199d5f5653cc85c70be6ed8adaf13"}
Nov 03 20:42:45.745612 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:45.745446    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds" event={"ID":"128a2ead-106d-483e-a4e0-8e13c7c91e28","Type":"ContainerStarted","Data":"ffcf83d4e1db5b4ef15c598023976af43d8a4cddcfd539b3af1724714dab0c6c"}
Nov 03 20:42:45.770405 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:45.770356    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds" podStartSLOduration=1.770343032 podStartE2EDuration="1.770343032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:45.76982831 +0000 UTC m=+3914.542121329" watchObservedRunningTime="2026-11-03 20:42:45.770343032 +0000 UTC m=+3914.542636046"
Nov 03 20:42:46.097886 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:46.096616    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:46.101348 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:46.101318    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-x4bds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body=
Nov 03 20:42:46.101480 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:46.101371    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds" podUID="128a2ead-106d-483e-a4e0-8e13c7c91e28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused"
Nov 03 20:42:47.100233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:47.097645    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-x4bds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body=
Nov 03 20:42:47.100233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:47.097715    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds" podUID="128a2ead-106d-483e-a4e0-8e13c7c91e28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused"
Nov 03 20:42:48.101093 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:48.100615    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-x4bds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body=
Nov 03 20:42:48.101093 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:48.100678    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds" podUID="128a2ead-106d-483e-a4e0-8e13c7c91e28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused"
Nov 03 20:42:49.101621 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:49.101584    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-x4bds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body=
Nov 03 20:42:49.102181 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:49.101641    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds" podUID="128a2ead-106d-483e-a4e0-8e13c7c91e28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused"
Nov 03 20:42:50.095677 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:50.095634    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-x4bds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body=
Nov 03 20:42:50.096006 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:50.095974    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds" podUID="128a2ead-106d-483e-a4e0-8e13c7c91e28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused"
Nov 03 20:42:51.096229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:51.095642    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-x4bds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body=
Nov 03 20:42:51.096229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:51.095728    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds" podUID="128a2ead-106d-483e-a4e0-8e13c7c91e28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused"
Nov 03 20:42:52.094576 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:52.094541    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-x4bds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body=
Nov 03 20:42:52.094770 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:52.094603    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds" podUID="128a2ead-106d-483e-a4e0-8e13c7c91e28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused"
Nov 03 20:42:53.695686 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:53.695652    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:53.696219 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:53.696012    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:53.697007 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:42:53.696985    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds"
Nov 03 20:42:58.723953 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:42:58.723912    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:43:10.999253 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:43:10.999193    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:43:15.411956 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:43:15.411922    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:43:25.839678 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:43:25.839640    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:43:28.762824 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:43:28.762782    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:43:41.825719 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:43:41.825632    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:43:45.441232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:43:45.435738    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:43:49.039039 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:43:49.039005    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:43:58.724971 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:43:58.724921    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:44:04.236189 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:44:04.236153    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:44:28.723880 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:44:28.723836    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:44:40.603349 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:44:40.603313    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:44:44.630386 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:44:44.630349    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:44:58.723630 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:44:58.723587    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:45:00.144371 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:00.144333    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-fhjxp"]
Nov 03 20:45:00.148070 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:00.148052    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-fhjxp"
Nov 03 20:45:00.155470 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:00.155374    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:45:00.155551 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:00.155472    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 20:45:00.155609 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:00.155548    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:45:00.157156 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:00.157134    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-fhjxp"]
Nov 03 20:45:00.273284 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:00.273239    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2f6a79-9045-4e17-8c28-ee413a239a25-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895645-fhjxp\" (UID: \"7d2f6a79-9045-4e17-8c28-ee413a239a25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-fhjxp"
Nov 03 20:45:00.273488 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:00.273367    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fz6d\" (UniqueName: \"kubernetes.io/projected/7d2f6a79-9045-4e17-8c28-ee413a239a25-kube-api-access-7fz6d\") pod \"maas-api-key-cleanup-29895645-fhjxp\" (UID: \"7d2f6a79-9045-4e17-8c28-ee413a239a25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-fhjxp"
Nov 03 20:45:00.374762 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:00.374721    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2f6a79-9045-4e17-8c28-ee413a239a25-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895645-fhjxp\" (UID: \"7d2f6a79-9045-4e17-8c28-ee413a239a25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-fhjxp"
Nov 03 20:45:00.374965 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:00.374803    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fz6d\" (UniqueName: \"kubernetes.io/projected/7d2f6a79-9045-4e17-8c28-ee413a239a25-kube-api-access-7fz6d\") pod \"maas-api-key-cleanup-29895645-fhjxp\" (UID: \"7d2f6a79-9045-4e17-8c28-ee413a239a25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-fhjxp"
Nov 03 20:45:00.375436 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:00.375414    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2f6a79-9045-4e17-8c28-ee413a239a25-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895645-fhjxp\" (UID: \"7d2f6a79-9045-4e17-8c28-ee413a239a25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-fhjxp"
Nov 03 20:45:00.387804 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:00.387775    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fz6d\" (UniqueName: \"kubernetes.io/projected/7d2f6a79-9045-4e17-8c28-ee413a239a25-kube-api-access-7fz6d\") pod \"maas-api-key-cleanup-29895645-fhjxp\" (UID: \"7d2f6a79-9045-4e17-8c28-ee413a239a25\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-fhjxp"
Nov 03 20:45:00.461910 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:00.461807    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-fhjxp"
Nov 03 20:45:00.737553 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:00.737497    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-fhjxp"]
Nov 03 20:45:00.740822 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:45:00.740792    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2f6a79_9045_4e17_8c28_ee413a239a25.slice/crio-4446bae2333263c1dc09ef38893458c3d1b593eade398d1d2f9a54279bcb5ac5 WatchSource:0}: Error finding container 4446bae2333263c1dc09ef38893458c3d1b593eade398d1d2f9a54279bcb5ac5: Status 404 returned error can't find the container with id 4446bae2333263c1dc09ef38893458c3d1b593eade398d1d2f9a54279bcb5ac5
Nov 03 20:45:01.612534 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:01.612495    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-fhjxp" event={"ID":"7d2f6a79-9045-4e17-8c28-ee413a239a25","Type":"ContainerStarted","Data":"950c1d029ace98f3216716e45160dfd9b16cc3afffead7f6ed60bd56f9ae22c7"}
Nov 03 20:45:01.613068 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:01.612556    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-fhjxp" event={"ID":"7d2f6a79-9045-4e17-8c28-ee413a239a25","Type":"ContainerStarted","Data":"4446bae2333263c1dc09ef38893458c3d1b593eade398d1d2f9a54279bcb5ac5"}
Nov 03 20:45:01.630955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:01.630897    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-fhjxp" podStartSLOduration=1.63087617 podStartE2EDuration="1.63087617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:01.630156418 +0000 UTC m=+4050.402449437" watchObservedRunningTime="2026-11-03 20:45:01.63087617 +0000 UTC m=+4050.403169190"
Nov 03 20:45:02.619184 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:02.619147    5591 generic.go:356] "Generic (PLEG): container finished" podID="7d2f6a79-9045-4e17-8c28-ee413a239a25" containerID="950c1d029ace98f3216716e45160dfd9b16cc3afffead7f6ed60bd56f9ae22c7" exitCode=60
Nov 03 20:45:02.619642 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:02.619235    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-fhjxp" event={"ID":"7d2f6a79-9045-4e17-8c28-ee413a239a25","Type":"ContainerDied","Data":"950c1d029ace98f3216716e45160dfd9b16cc3afffead7f6ed60bd56f9ae22c7"}
Nov 03 20:45:02.619642 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:02.619520    5591 scope.go:122] "RemoveContainer" containerID="950c1d029ace98f3216716e45160dfd9b16cc3afffead7f6ed60bd56f9ae22c7"
Nov 03 20:45:03.626052 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:03.626011    5591 generic.go:356] "Generic (PLEG): container finished" podID="7d2f6a79-9045-4e17-8c28-ee413a239a25" containerID="20fef177eaa656584301db106fb8e463c55b1f11c57c8176dfc89ad3b6e7842e" exitCode=60
Nov 03 20:45:03.626549 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:03.626071    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-fhjxp" event={"ID":"7d2f6a79-9045-4e17-8c28-ee413a239a25","Type":"ContainerDied","Data":"20fef177eaa656584301db106fb8e463c55b1f11c57c8176dfc89ad3b6e7842e"}
Nov 03 20:45:03.626549 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:03.626140    5591 scope.go:122] "RemoveContainer" containerID="950c1d029ace98f3216716e45160dfd9b16cc3afffead7f6ed60bd56f9ae22c7"
Nov 03 20:45:03.626549 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:03.626383    5591 scope.go:122] "RemoveContainer" containerID="20fef177eaa656584301db106fb8e463c55b1f11c57c8176dfc89ad3b6e7842e"
Nov 03 20:45:03.626698 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:45:03.626667    5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895645-fhjxp_odh-ai-gateway-infra(7d2f6a79-9045-4e17-8c28-ee413a239a25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-fhjxp" podUID="7d2f6a79-9045-4e17-8c28-ee413a239a25"
Nov 03 20:45:04.632543 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:04.632518    5591 scope.go:122] "RemoveContainer" containerID="20fef177eaa656584301db106fb8e463c55b1f11c57c8176dfc89ad3b6e7842e"
Nov 03 20:45:04.632980 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:45:04.632747    5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895645-fhjxp_odh-ai-gateway-infra(7d2f6a79-9045-4e17-8c28-ee413a239a25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-fhjxp" podUID="7d2f6a79-9045-4e17-8c28-ee413a239a25"
Nov 03 20:45:15.341952 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:15.341855    5591 scope.go:122] "RemoveContainer" containerID="20fef177eaa656584301db106fb8e463c55b1f11c57c8176dfc89ad3b6e7842e"
Nov 03 20:45:16.691732 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:16.691695    5591 generic.go:356] "Generic (PLEG): container finished" podID="7d2f6a79-9045-4e17-8c28-ee413a239a25" containerID="0a86019c1c6fe81a3c666dc61444ee97b0ee612ece7e73b0b9cb879ceb627eaa" exitCode=60
Nov 03 20:45:16.692186 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:16.691768    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-fhjxp" event={"ID":"7d2f6a79-9045-4e17-8c28-ee413a239a25","Type":"ContainerDied","Data":"0a86019c1c6fe81a3c666dc61444ee97b0ee612ece7e73b0b9cb879ceb627eaa"}
Nov 03 20:45:16.692186 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:16.691833    5591 scope.go:122] "RemoveContainer" containerID="20fef177eaa656584301db106fb8e463c55b1f11c57c8176dfc89ad3b6e7842e"
Nov 03 20:45:16.692186 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:16.692162    5591 scope.go:122] "RemoveContainer" containerID="0a86019c1c6fe81a3c666dc61444ee97b0ee612ece7e73b0b9cb879ceb627eaa"
Nov 03 20:45:16.692451 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:45:16.692428    5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895645-fhjxp_odh-ai-gateway-infra(7d2f6a79-9045-4e17-8c28-ee413a239a25)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-fhjxp" podUID="7d2f6a79-9045-4e17-8c28-ee413a239a25"
Nov 03 20:45:17.719555 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:17.719519    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-fhjxp"]
Nov 03 20:45:17.859919 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:17.859894    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-fhjxp"
Nov 03 20:45:17.952373 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:17.952339    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d2f6a79-9045-4e17-8c28-ee413a239a25-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2f6a79-9045-4e17-8c28-ee413a239a25-maas-api-service-ca\") pod \"7d2f6a79-9045-4e17-8c28-ee413a239a25\" (UID: \"7d2f6a79-9045-4e17-8c28-ee413a239a25\") "
Nov 03 20:45:17.952559 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:17.952478    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d2f6a79-9045-4e17-8c28-ee413a239a25-kube-api-access-7fz6d\" (UniqueName: \"kubernetes.io/projected/7d2f6a79-9045-4e17-8c28-ee413a239a25-kube-api-access-7fz6d\") pod \"7d2f6a79-9045-4e17-8c28-ee413a239a25\" (UID: \"7d2f6a79-9045-4e17-8c28-ee413a239a25\") "
Nov 03 20:45:17.952773 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:17.952752    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2f6a79-9045-4e17-8c28-ee413a239a25-maas-api-service-ca" pod "7d2f6a79-9045-4e17-8c28-ee413a239a25" (UID: "7d2f6a79-9045-4e17-8c28-ee413a239a25"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:17.952889 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:17.952875    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2f6a79-9045-4e17-8c28-ee413a239a25-maas-api-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:17.954603 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:17.954575    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2f6a79-9045-4e17-8c28-ee413a239a25-kube-api-access-7fz6d" pod "7d2f6a79-9045-4e17-8c28-ee413a239a25" (UID: "7d2f6a79-9045-4e17-8c28-ee413a239a25"). InnerVolumeSpecName "kube-api-access-7fz6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:18.054356 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:18.054319    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fz6d\" (UniqueName: \"kubernetes.io/projected/7d2f6a79-9045-4e17-8c28-ee413a239a25-kube-api-access-7fz6d\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:18.706222 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:18.706161    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-fhjxp" event={"ID":"7d2f6a79-9045-4e17-8c28-ee413a239a25","Type":"ContainerDied","Data":"4446bae2333263c1dc09ef38893458c3d1b593eade398d1d2f9a54279bcb5ac5"}
Nov 03 20:45:18.706222 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:18.706222    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895645-fhjxp"
Nov 03 20:45:18.706458 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:18.706258    5591 scope.go:122] "RemoveContainer" containerID="0a86019c1c6fe81a3c666dc61444ee97b0ee612ece7e73b0b9cb879ceb627eaa"
Nov 03 20:45:18.730323 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:18.730291    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-fhjxp"]
Nov 03 20:45:18.738247 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:18.738219    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895645-fhjxp"]
Nov 03 20:45:19.348001 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:19.347970    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d2f6a79-9045-4e17-8c28-ee413a239a25" path="/var/lib/kubelet/pods/7d2f6a79-9045-4e17-8c28-ee413a239a25/volumes"
Nov 03 20:45:24.705793 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:24.705760    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:45:28.723534 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:45:28.723494    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:45:32.961737 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:32.961704    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3953989c-x4ms7"]
Nov 03 20:45:32.962299 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:32.962132    5591 cpu_manager.go:414] "containerMap: removing container" podUID="7d2f6a79-9045-4e17-8c28-ee413a239a25" containerName="cleanup"
Nov 03 20:45:32.962299 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:32.962144    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d2f6a79-9045-4e17-8c28-ee413a239a25" containerName="cleanup"
Nov 03 20:45:32.962299 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:32.962158    5591 cpu_manager.go:414] "containerMap: removing container" podUID="7d2f6a79-9045-4e17-8c28-ee413a239a25" containerName="cleanup"
Nov 03 20:45:32.962299 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:32.962163    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d2f6a79-9045-4e17-8c28-ee413a239a25" containerName="cleanup"
Nov 03 20:45:32.962299 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:32.962177    5591 cpu_manager.go:414] "containerMap: removing container" podUID="7d2f6a79-9045-4e17-8c28-ee413a239a25" containerName="cleanup"
Nov 03 20:45:32.962299 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:32.962181    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d2f6a79-9045-4e17-8c28-ee413a239a25" containerName="cleanup"
Nov 03 20:45:32.962299 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:32.962259    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d2f6a79-9045-4e17-8c28-ee413a239a25" containerName="cleanup"
Nov 03 20:45:32.962299 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:32.962274    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d2f6a79-9045-4e17-8c28-ee413a239a25" containerName="cleanup"
Nov 03 20:45:32.966689 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:32.966668    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3953989c-x4ms7"
Nov 03 20:45:32.970419 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:32.970394    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 20:45:32.973905 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:32.973886    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:45:32.974435 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:32.974359    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:45:32.976942 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:32.976905    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3953989c-x4ms7"]
Nov 03 20:45:33.015343 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:33.015300    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk8q2\" (UniqueName: \"kubernetes.io/projected/43c8ac25-77ff-4dc1-ae09-050cd7e4504a-kube-api-access-kk8q2\") pod \"maas-api-revoke-sub-3953989c-x4ms7\" (UID: \"43c8ac25-77ff-4dc1-ae09-050cd7e4504a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3953989c-x4ms7"
Nov 03 20:45:33.015523 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:33.015416    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c8ac25-77ff-4dc1-ae09-050cd7e4504a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3953989c-x4ms7\" (UID: \"43c8ac25-77ff-4dc1-ae09-050cd7e4504a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3953989c-x4ms7"
Nov 03 20:45:33.015523 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:33.015450    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c8ac25-77ff-4dc1-ae09-050cd7e4504a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3953989c-x4ms7\" (UID: \"43c8ac25-77ff-4dc1-ae09-050cd7e4504a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3953989c-x4ms7"
Nov 03 20:45:33.070594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:33.070564    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:45:33.116332 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:33.116291    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk8q2\" (UniqueName: \"kubernetes.io/projected/43c8ac25-77ff-4dc1-ae09-050cd7e4504a-kube-api-access-kk8q2\") pod \"maas-api-revoke-sub-3953989c-x4ms7\" (UID: \"43c8ac25-77ff-4dc1-ae09-050cd7e4504a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3953989c-x4ms7"
Nov 03 20:45:33.116510 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:33.116404    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c8ac25-77ff-4dc1-ae09-050cd7e4504a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3953989c-x4ms7\" (UID: \"43c8ac25-77ff-4dc1-ae09-050cd7e4504a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3953989c-x4ms7"
Nov 03 20:45:33.116510 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:33.116445    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c8ac25-77ff-4dc1-ae09-050cd7e4504a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3953989c-x4ms7\" (UID: \"43c8ac25-77ff-4dc1-ae09-050cd7e4504a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3953989c-x4ms7"
Nov 03 20:45:33.117122 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:33.117101    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c8ac25-77ff-4dc1-ae09-050cd7e4504a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3953989c-x4ms7\" (UID: \"43c8ac25-77ff-4dc1-ae09-050cd7e4504a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3953989c-x4ms7"
Nov 03 20:45:33.118964 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:33.118940    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c8ac25-77ff-4dc1-ae09-050cd7e4504a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3953989c-x4ms7\" (UID: \"43c8ac25-77ff-4dc1-ae09-050cd7e4504a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3953989c-x4ms7"
Nov 03 20:45:33.129094 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:33.129072    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk8q2\" (UniqueName: \"kubernetes.io/projected/43c8ac25-77ff-4dc1-ae09-050cd7e4504a-kube-api-access-kk8q2\") pod \"maas-api-revoke-sub-3953989c-x4ms7\" (UID: \"43c8ac25-77ff-4dc1-ae09-050cd7e4504a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3953989c-x4ms7"
Nov 03 20:45:33.281081 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:33.281055    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3953989c-x4ms7"
Nov 03 20:45:33.557540 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:33.557463    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3953989c-x4ms7"]
Nov 03 20:45:33.560613 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:45:33.560580    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c8ac25_77ff_4dc1_ae09_050cd7e4504a.slice/crio-14a980c82559fdb150091683e70be98e59cfc6ff7a1ac37d3b5ec3bd841be326 WatchSource:0}: Error finding container 14a980c82559fdb150091683e70be98e59cfc6ff7a1ac37d3b5ec3bd841be326: Status 404 returned error can't find the container with id 14a980c82559fdb150091683e70be98e59cfc6ff7a1ac37d3b5ec3bd841be326
Nov 03 20:45:33.796018 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:33.795987    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3953989c-x4ms7" event={"ID":"43c8ac25-77ff-4dc1-ae09-050cd7e4504a","Type":"ContainerStarted","Data":"14a980c82559fdb150091683e70be98e59cfc6ff7a1ac37d3b5ec3bd841be326"}
Nov 03 20:45:34.802892 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:34.802861    5591 generic.go:356] "Generic (PLEG): container finished" podID="43c8ac25-77ff-4dc1-ae09-050cd7e4504a" containerID="c7b5e7ca8af5dbc5d03d508aa7a06fbf71a53382779111f8d320b9eaedd68797" exitCode=0
Nov 03 20:45:34.803399 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:34.802948    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3953989c-x4ms7" event={"ID":"43c8ac25-77ff-4dc1-ae09-050cd7e4504a","Type":"ContainerDied","Data":"c7b5e7ca8af5dbc5d03d508aa7a06fbf71a53382779111f8d320b9eaedd68797"}
Nov 03 20:45:35.941824 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:35.941798    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3953989c-x4ms7"
Nov 03 20:45:36.044170 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:36.044134    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43c8ac25-77ff-4dc1-ae09-050cd7e4504a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c8ac25-77ff-4dc1-ae09-050cd7e4504a-maas-api-xks-ca\") pod \"43c8ac25-77ff-4dc1-ae09-050cd7e4504a\" (UID: \"43c8ac25-77ff-4dc1-ae09-050cd7e4504a\") "
Nov 03 20:45:36.044413 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:36.044274    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43c8ac25-77ff-4dc1-ae09-050cd7e4504a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c8ac25-77ff-4dc1-ae09-050cd7e4504a-maas-api-openshift-ca\") pod \"43c8ac25-77ff-4dc1-ae09-050cd7e4504a\" (UID: \"43c8ac25-77ff-4dc1-ae09-050cd7e4504a\") "
Nov 03 20:45:36.044413 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:36.044387    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c8ac25-77ff-4dc1-ae09-050cd7e4504a-kube-api-access-kk8q2\" (UniqueName: \"kubernetes.io/projected/43c8ac25-77ff-4dc1-ae09-050cd7e4504a-kube-api-access-kk8q2\") pod \"43c8ac25-77ff-4dc1-ae09-050cd7e4504a\" (UID: \"43c8ac25-77ff-4dc1-ae09-050cd7e4504a\") "
Nov 03 20:45:36.044632 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:36.044607    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c8ac25-77ff-4dc1-ae09-050cd7e4504a-maas-api-openshift-ca" pod "43c8ac25-77ff-4dc1-ae09-050cd7e4504a" (UID: "43c8ac25-77ff-4dc1-ae09-050cd7e4504a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:45:36.044757 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:36.044739    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43c8ac25-77ff-4dc1-ae09-050cd7e4504a-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:36.046476 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:36.046449    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c8ac25-77ff-4dc1-ae09-050cd7e4504a-kube-api-access-kk8q2" pod "43c8ac25-77ff-4dc1-ae09-050cd7e4504a" (UID: "43c8ac25-77ff-4dc1-ae09-050cd7e4504a"). InnerVolumeSpecName "kube-api-access-kk8q2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:45:36.046544 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:36.046522    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c8ac25-77ff-4dc1-ae09-050cd7e4504a-maas-api-xks-ca" pod "43c8ac25-77ff-4dc1-ae09-050cd7e4504a" (UID: "43c8ac25-77ff-4dc1-ae09-050cd7e4504a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:45:36.146018 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:36.145931    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk8q2\" (UniqueName: \"kubernetes.io/projected/43c8ac25-77ff-4dc1-ae09-050cd7e4504a-kube-api-access-kk8q2\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:36.146018 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:36.145964    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43c8ac25-77ff-4dc1-ae09-050cd7e4504a-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:45:36.814184 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:36.814153    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3953989c-x4ms7"
Nov 03 20:45:36.814412 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:36.814179    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3953989c-x4ms7" event={"ID":"43c8ac25-77ff-4dc1-ae09-050cd7e4504a","Type":"ContainerDied","Data":"14a980c82559fdb150091683e70be98e59cfc6ff7a1ac37d3b5ec3bd841be326"}
Nov 03 20:45:36.814412 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:36.814261    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14a980c82559fdb150091683e70be98e59cfc6ff7a1ac37d3b5ec3bd841be326"
Nov 03 20:45:43.915144 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:43.915106    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:45:48.045831 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:48.045795    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:45:57.632898 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:57.632853    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4da46e29-vm69x"]
Nov 03 20:45:57.633546 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:57.633523    5591 cpu_manager.go:414] "containerMap: removing container" podUID="43c8ac25-77ff-4dc1-ae09-050cd7e4504a" containerName="revoke-keys"
Nov 03 20:45:57.633639 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:57.633548    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c8ac25-77ff-4dc1-ae09-050cd7e4504a" containerName="revoke-keys"
Nov 03 20:45:57.633710 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:57.633652    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c8ac25-77ff-4dc1-ae09-050cd7e4504a" containerName="revoke-keys"
Nov 03 20:45:57.633710 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:57.633672    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d2f6a79-9045-4e17-8c28-ee413a239a25" containerName="cleanup"
Nov 03 20:45:57.638135 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:57.638113    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da46e29-vm69x"
Nov 03 20:45:57.646857 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:57.645685    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:45:57.646965 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:57.645802    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:45:57.646965 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:57.646032    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:45:57.648964 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:57.648914    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4da46e29-vm69x"]
Nov 03 20:45:57.649255 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:57.649234    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 20:45:57.773642 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:57.773597    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e06afc0a-1823-4700-b5be-42206eb9ecb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4da46e29-vm69x\" (UID: \"e06afc0a-1823-4700-b5be-42206eb9ecb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da46e29-vm69x"
Nov 03 20:45:57.773897 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:57.773783    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e06afc0a-1823-4700-b5be-42206eb9ecb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4da46e29-vm69x\" (UID: \"e06afc0a-1823-4700-b5be-42206eb9ecb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da46e29-vm69x"
Nov 03 20:45:57.773897 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:57.773815    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcfwd\" (UniqueName: \"kubernetes.io/projected/e06afc0a-1823-4700-b5be-42206eb9ecb8-kube-api-access-gcfwd\") pod \"maas-api-revoke-sub-4da46e29-vm69x\" (UID: \"e06afc0a-1823-4700-b5be-42206eb9ecb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da46e29-vm69x"
Nov 03 20:45:57.874686 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:57.874643    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e06afc0a-1823-4700-b5be-42206eb9ecb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4da46e29-vm69x\" (UID: \"e06afc0a-1823-4700-b5be-42206eb9ecb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da46e29-vm69x"
Nov 03 20:45:57.874686 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:57.874684    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcfwd\" (UniqueName: \"kubernetes.io/projected/e06afc0a-1823-4700-b5be-42206eb9ecb8-kube-api-access-gcfwd\") pod \"maas-api-revoke-sub-4da46e29-vm69x\" (UID: \"e06afc0a-1823-4700-b5be-42206eb9ecb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da46e29-vm69x"
Nov 03 20:45:57.874996 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:57.874733    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e06afc0a-1823-4700-b5be-42206eb9ecb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4da46e29-vm69x\" (UID: \"e06afc0a-1823-4700-b5be-42206eb9ecb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da46e29-vm69x"
Nov 03 20:45:57.875672 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:57.875642    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e06afc0a-1823-4700-b5be-42206eb9ecb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4da46e29-vm69x\" (UID: \"e06afc0a-1823-4700-b5be-42206eb9ecb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da46e29-vm69x"
Nov 03 20:45:57.877397 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:57.877377    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e06afc0a-1823-4700-b5be-42206eb9ecb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4da46e29-vm69x\" (UID: \"e06afc0a-1823-4700-b5be-42206eb9ecb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da46e29-vm69x"
Nov 03 20:45:57.887406 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:57.887347    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcfwd\" (UniqueName: \"kubernetes.io/projected/e06afc0a-1823-4700-b5be-42206eb9ecb8-kube-api-access-gcfwd\") pod \"maas-api-revoke-sub-4da46e29-vm69x\" (UID: \"e06afc0a-1823-4700-b5be-42206eb9ecb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da46e29-vm69x"
Nov 03 20:45:57.954354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:57.954317    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da46e29-vm69x"
Nov 03 20:45:58.251603 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:58.251571    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4da46e29-vm69x"]
Nov 03 20:45:58.254069 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:45:58.254028    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode06afc0a_1823_4700_b5be_42206eb9ecb8.slice/crio-a5e1fc49e3a5c4d9ca2ff5b69e373554ac63d69a3132eccecb414c8e21666d97 WatchSource:0}: Error finding container a5e1fc49e3a5c4d9ca2ff5b69e373554ac63d69a3132eccecb414c8e21666d97: Status 404 returned error can't find the container with id a5e1fc49e3a5c4d9ca2ff5b69e373554ac63d69a3132eccecb414c8e21666d97
Nov 03 20:45:58.773112 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:45:58.773073    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:45:58.948740 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:58.948295    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da46e29-vm69x" event={"ID":"e06afc0a-1823-4700-b5be-42206eb9ecb8","Type":"ContainerStarted","Data":"a98e498892de19189bdbea1f69ba06885df6c56f2e2b856c9a8b6f00b8dba97e"}
Nov 03 20:45:58.948740 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:58.948365    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da46e29-vm69x" event={"ID":"e06afc0a-1823-4700-b5be-42206eb9ecb8","Type":"ContainerStarted","Data":"a5e1fc49e3a5c4d9ca2ff5b69e373554ac63d69a3132eccecb414c8e21666d97"}
Nov 03 20:45:58.968477 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:58.968423    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da46e29-vm69x" podStartSLOduration=1.968408267 podStartE2EDuration="1.968408267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:58.967076818 +0000 UTC m=+4107.739369837" watchObservedRunningTime="2026-11-03 20:45:58.968408267 +0000 UTC m=+4107.740701285"
Nov 03 20:45:59.957793 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:59.957756    5591 generic.go:356] "Generic (PLEG): container finished" podID="e06afc0a-1823-4700-b5be-42206eb9ecb8" containerID="a98e498892de19189bdbea1f69ba06885df6c56f2e2b856c9a8b6f00b8dba97e" exitCode=0
Nov 03 20:45:59.958249 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:45:59.957834    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da46e29-vm69x" event={"ID":"e06afc0a-1823-4700-b5be-42206eb9ecb8","Type":"ContainerDied","Data":"a98e498892de19189bdbea1f69ba06885df6c56f2e2b856c9a8b6f00b8dba97e"}
Nov 03 20:46:01.110920 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:01.110893    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da46e29-vm69x"
Nov 03 20:46:01.259993 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:01.259898    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e06afc0a-1823-4700-b5be-42206eb9ecb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e06afc0a-1823-4700-b5be-42206eb9ecb8-maas-api-xks-ca\") pod \"e06afc0a-1823-4700-b5be-42206eb9ecb8\" (UID: \"e06afc0a-1823-4700-b5be-42206eb9ecb8\") "
Nov 03 20:46:01.259993 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:01.259946    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e06afc0a-1823-4700-b5be-42206eb9ecb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e06afc0a-1823-4700-b5be-42206eb9ecb8-maas-api-openshift-ca\") pod \"e06afc0a-1823-4700-b5be-42206eb9ecb8\" (UID: \"e06afc0a-1823-4700-b5be-42206eb9ecb8\") "
Nov 03 20:46:01.259993 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:01.259982    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e06afc0a-1823-4700-b5be-42206eb9ecb8-kube-api-access-gcfwd\" (UniqueName: \"kubernetes.io/projected/e06afc0a-1823-4700-b5be-42206eb9ecb8-kube-api-access-gcfwd\") pod \"e06afc0a-1823-4700-b5be-42206eb9ecb8\" (UID: \"e06afc0a-1823-4700-b5be-42206eb9ecb8\") "
Nov 03 20:46:01.260374 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:01.260349    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e06afc0a-1823-4700-b5be-42206eb9ecb8-maas-api-openshift-ca" pod "e06afc0a-1823-4700-b5be-42206eb9ecb8" (UID: "e06afc0a-1823-4700-b5be-42206eb9ecb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:46:01.262291 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:01.262269    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06afc0a-1823-4700-b5be-42206eb9ecb8-kube-api-access-gcfwd" pod "e06afc0a-1823-4700-b5be-42206eb9ecb8" (UID: "e06afc0a-1823-4700-b5be-42206eb9ecb8"). InnerVolumeSpecName "kube-api-access-gcfwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:46:01.262374 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:01.262314    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e06afc0a-1823-4700-b5be-42206eb9ecb8-maas-api-xks-ca" pod "e06afc0a-1823-4700-b5be-42206eb9ecb8" (UID: "e06afc0a-1823-4700-b5be-42206eb9ecb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:46:01.361087 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:01.361060    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e06afc0a-1823-4700-b5be-42206eb9ecb8-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:01.361087 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:01.361086    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e06afc0a-1823-4700-b5be-42206eb9ecb8-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:01.361290 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:01.361095    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcfwd\" (UniqueName: \"kubernetes.io/projected/e06afc0a-1823-4700-b5be-42206eb9ecb8-kube-api-access-gcfwd\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:01.969569 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:01.969540    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da46e29-vm69x"
Nov 03 20:46:01.969569 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:01.969562    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4da46e29-vm69x" event={"ID":"e06afc0a-1823-4700-b5be-42206eb9ecb8","Type":"ContainerDied","Data":"a5e1fc49e3a5c4d9ca2ff5b69e373554ac63d69a3132eccecb414c8e21666d97"}
Nov 03 20:46:01.969806 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:01.969610    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5e1fc49e3a5c4d9ca2ff5b69e373554ac63d69a3132eccecb414c8e21666d97"
Nov 03 20:46:09.179175 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:09.179133    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"]
Nov 03 20:46:09.179683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:09.179379    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts" podUID="8f55488b-ab5f-4da2-bbbe-ef8c0fa41882" containerName="discovery" containerID="cri-o://38c819eb177ad2bdeaed4a0680eb37f29014c680eb5fd1602ee4660db56159ae" gracePeriod=30
Nov 03 20:46:09.437265 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:09.437177    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"
Nov 03 20:46:09.540816 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:09.540783    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-csr-dns-cert\") pod \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") "
Nov 03 20:46:09.541001 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:09.540850    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-kube-api-access-k46j4\" (UniqueName: \"kubernetes.io/projected/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-kube-api-access-k46j4\") pod \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") "
Nov 03 20:46:09.541001 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:09.540876    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-kubeconfig\") pod \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") "
Nov 03 20:46:09.541001 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:09.540918    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-csr-ca-configmap\") pod \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") "
Nov 03 20:46:09.541151 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:09.541060    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-local-certs\") pod \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") "
Nov 03 20:46:09.541151 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:09.541121    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-cacerts\" (UniqueName: \"kubernetes.io/secret/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-cacerts\") pod \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") "
Nov 03 20:46:09.541304 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:09.541160    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-token\" (UniqueName: \"kubernetes.io/projected/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-token\") pod \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\" (UID: \"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882\") "
Nov 03 20:46:09.541390 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:09.541353    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-csr-ca-configmap" pod "8f55488b-ab5f-4da2-bbbe-ef8c0fa41882" (UID: "8f55488b-ab5f-4da2-bbbe-ef8c0fa41882"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:46:09.541570 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:09.541545    5591 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-csr-ca-configmap\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:09.543429 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:09.543404    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-kube-api-access-k46j4" pod "8f55488b-ab5f-4da2-bbbe-ef8c0fa41882" (UID: "8f55488b-ab5f-4da2-bbbe-ef8c0fa41882"). InnerVolumeSpecName "kube-api-access-k46j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:46:09.543566 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:09.543405    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-kubeconfig" pod "8f55488b-ab5f-4da2-bbbe-ef8c0fa41882" (UID: "8f55488b-ab5f-4da2-bbbe-ef8c0fa41882"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:46:09.543805 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:09.543783    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-local-certs" pod "8f55488b-ab5f-4da2-bbbe-ef8c0fa41882" (UID: "8f55488b-ab5f-4da2-bbbe-ef8c0fa41882"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:46:09.543892 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:09.543869    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-token" pod "8f55488b-ab5f-4da2-bbbe-ef8c0fa41882" (UID: "8f55488b-ab5f-4da2-bbbe-ef8c0fa41882"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:46:09.543948 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:09.543878    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-csr-dns-cert" pod "8f55488b-ab5f-4da2-bbbe-ef8c0fa41882" (UID: "8f55488b-ab5f-4da2-bbbe-ef8c0fa41882"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:46:09.543948 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:09.543928    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-cacerts" pod "8f55488b-ab5f-4da2-bbbe-ef8c0fa41882" (UID: "8f55488b-ab5f-4da2-bbbe-ef8c0fa41882"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:46:09.642682 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:09.642645    5591 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-local-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:09.642682 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:09.642674    5591 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-cacerts\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:09.642682 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:09.642684    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:09.642940 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:09.642694    5591 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-csr-dns-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:09.642940 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:09.642704    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k46j4\" (UniqueName: \"kubernetes.io/projected/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-kube-api-access-k46j4\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:09.642940 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:09.642713    5591 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882-istio-kubeconfig\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:46:10.010913 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.010876    5591 generic.go:356] "Generic (PLEG): container finished" podID="8f55488b-ab5f-4da2-bbbe-ef8c0fa41882" containerID="38c819eb177ad2bdeaed4a0680eb37f29014c680eb5fd1602ee4660db56159ae" exitCode=0
Nov 03 20:46:10.011111 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.010923    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"
Nov 03 20:46:10.011111 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.010969    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts" event={"ID":"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882","Type":"ContainerDied","Data":"38c819eb177ad2bdeaed4a0680eb37f29014c680eb5fd1602ee4660db56159ae"}
Nov 03 20:46:10.011111 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.011018    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts" event={"ID":"8f55488b-ab5f-4da2-bbbe-ef8c0fa41882","Type":"ContainerDied","Data":"caa1f821f2693c9210cbd02beeea9926a92e5007449999784894a36399862fed"}
Nov 03 20:46:10.011111 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.011040    5591 scope.go:122] "RemoveContainer" containerID="38c819eb177ad2bdeaed4a0680eb37f29014c680eb5fd1602ee4660db56159ae"
Nov 03 20:46:10.033967 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.033945    5591 scope.go:122] "RemoveContainer" containerID="38c819eb177ad2bdeaed4a0680eb37f29014c680eb5fd1602ee4660db56159ae"
Nov 03 20:46:10.034810 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:46:10.034743    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38c819eb177ad2bdeaed4a0680eb37f29014c680eb5fd1602ee4660db56159ae\": container with ID starting with 38c819eb177ad2bdeaed4a0680eb37f29014c680eb5fd1602ee4660db56159ae not found: ID does not exist" containerID="38c819eb177ad2bdeaed4a0680eb37f29014c680eb5fd1602ee4660db56159ae"
Nov 03 20:46:10.034810 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.034785    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38c819eb177ad2bdeaed4a0680eb37f29014c680eb5fd1602ee4660db56159ae"} err="failed to get container status \"38c819eb177ad2bdeaed4a0680eb37f29014c680eb5fd1602ee4660db56159ae\": rpc error: code = NotFound desc = could not find container \"38c819eb177ad2bdeaed4a0680eb37f29014c680eb5fd1602ee4660db56159ae\": container with ID starting with 38c819eb177ad2bdeaed4a0680eb37f29014c680eb5fd1602ee4660db56159ae not found: ID does not exist"
Nov 03 20:46:10.036192 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.036173    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"]
Nov 03 20:46:10.040176 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.040152    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xr6ts"]
Nov 03 20:46:10.220674 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.220634    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"]
Nov 03 20:46:10.221376 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.221357    5591 cpu_manager.go:414] "containerMap: removing container" podUID="e06afc0a-1823-4700-b5be-42206eb9ecb8" containerName="revoke-keys"
Nov 03 20:46:10.221434 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.221379    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e06afc0a-1823-4700-b5be-42206eb9ecb8" containerName="revoke-keys"
Nov 03 20:46:10.221434 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.221406    5591 cpu_manager.go:414] "containerMap: removing container" podUID="8f55488b-ab5f-4da2-bbbe-ef8c0fa41882" containerName="discovery"
Nov 03 20:46:10.221434 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.221414    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f55488b-ab5f-4da2-bbbe-ef8c0fa41882" containerName="discovery"
Nov 03 20:46:10.221539 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.221507    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f55488b-ab5f-4da2-bbbe-ef8c0fa41882" containerName="discovery"
Nov 03 20:46:10.221539 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.221529    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="e06afc0a-1823-4700-b5be-42206eb9ecb8" containerName="revoke-keys"
Nov 03 20:46:10.226006 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.225987    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.236831 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.236809    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"]
Nov 03 20:46:10.241714 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.241693    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-e7bf5b-openshift-default-dockercfg-c9d7m\""
Nov 03 20:46:10.249310 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.249283    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-workload-certs\") pod \"e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") " pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.249452 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.249325    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjxz7\" (UniqueName: \"kubernetes.io/projected/1b577df4-3ef8-4c04-8f9e-e454254267d6-kube-api-access-kjxz7\") pod \"e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") " pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.249536 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.249475    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-podinfo\") pod \"e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") " pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.249598 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.249541    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b577df4-3ef8-4c04-8f9e-e454254267d6-istiod-ca-cert\") pod \"e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") " pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.249659 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.249610    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-envoy\") pod \"e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") " pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.249659 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.249644    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-credential-socket\") pod \"e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") " pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.249781 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.249767    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-workload-socket\") pod \"e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") " pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.249818 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.249794    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-token\") pod \"e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") " pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.249866 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.249814    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-data\") pod \"e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") " pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.350890 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.350853    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-podinfo\") pod \"e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") " pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.350890 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.350889    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b577df4-3ef8-4c04-8f9e-e454254267d6-istiod-ca-cert\") pod \"e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") " pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.351193 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.350929    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-envoy\") pod \"e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") " pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.351193 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.350946    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-credential-socket\") pod \"e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") " pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.351193 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.350993    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-workload-socket\") pod \"e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") " pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.351193 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.351025    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-token\") pod \"e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") " pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.351193 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.351049    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-data\") pod \"e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") " pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.351193 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.351095    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-workload-certs\") pod \"e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") " pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.351193 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.351116    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjxz7\" (UniqueName: \"kubernetes.io/projected/1b577df4-3ef8-4c04-8f9e-e454254267d6-kube-api-access-kjxz7\") pod \"e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") " pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.351687 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.351641    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-data\") pod \"e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") " pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.351887 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.351856    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-credential-socket\") pod \"e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") " pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.351983 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.351896    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-workload-socket\") pod \"e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") " pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.352076 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.352031    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b577df4-3ef8-4c04-8f9e-e454254267d6-istiod-ca-cert\") pod \"e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") " pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.352283 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.352263    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-workload-certs\") pod \"e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") " pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.354119 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.354097    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-envoy\") pod \"e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") " pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.354313 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.354291    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-podinfo\") pod \"e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") " pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.364703 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.364662    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-token\") pod \"e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") " pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.364897 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.364786    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjxz7\" (UniqueName: \"kubernetes.io/projected/1b577df4-3ef8-4c04-8f9e-e454254267d6-kube-api-access-kjxz7\") pod \"e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") " pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.544843 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.544806    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:10.834445 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.834410    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"]
Nov 03 20:46:10.843743 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.843698    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:46:10.843846 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.843786    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:46:10.843846 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:10.843825    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:46:11.018815 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:11.018777    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk" event={"ID":"1b577df4-3ef8-4c04-8f9e-e454254267d6","Type":"ContainerStarted","Data":"95554c60f1c41cbc6106ca2be125586771e9ba47098c88d2bbb7cf4273ed8d31"}
Nov 03 20:46:11.019024 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:11.018837    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk" event={"ID":"1b577df4-3ef8-4c04-8f9e-e454254267d6","Type":"ContainerStarted","Data":"06af1692fc8f20764de54d6091db91f31e590aa63924485347dbc52c824d76b1"}
Nov 03 20:46:11.043704 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:11.043641    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk" podStartSLOduration=1.0436224649999999 podStartE2EDuration="1.043622465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:11.041083093 +0000 UTC m=+4119.813376198" watchObservedRunningTime="2026-11-03 20:46:11.043622465 +0000 UTC m=+4119.815915487"
Nov 03 20:46:11.349103 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:11.349073    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f55488b-ab5f-4da2-bbbe-ef8c0fa41882" path="/var/lib/kubelet/pods/8f55488b-ab5f-4da2-bbbe-ef8c0fa41882/volumes"
Nov 03 20:46:11.545858 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:11.545815    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:11.551142 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:11.551115    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:12.025882 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:12.025848    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:12.026922 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:12.026900    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:24.464377 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.464318    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs"]
Nov 03 20:46:24.469302 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.469275    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs"]
Nov 03 20:46:24.469481 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.469402    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs"
Nov 03 20:46:24.476056 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.476030    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:46:24.476233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.476070    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:46:24.481563 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.481538    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 20:46:24.481563 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.481545    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-942q2\""
Nov 03 20:46:24.481738 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.481541    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-e7bf5b-serving-cert\""
Nov 03 20:46:24.573415 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.573385    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9"]
Nov 03 20:46:24.577105 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.577088    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9"
Nov 03 20:46:24.585818 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.585792    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-e7bf5b\""
Nov 03 20:46:24.586004 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.585902    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-4db0eb42-dockercfg-dvw9j\""
Nov 03 20:46:24.587602 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.587576    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9"]
Nov 03 20:46:24.590818 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.590795    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f02f361-0848-4bda-b2f0-2301110d1b1f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs\" (UID: \"8f02f361-0848-4bda-b2f0-2301110d1b1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs"
Nov 03 20:46:24.590934 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.590833    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75c9h\" (UniqueName: \"kubernetes.io/projected/8f02f361-0848-4bda-b2f0-2301110d1b1f-kube-api-access-75c9h\") pod \"maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs\" (UID: \"8f02f361-0848-4bda-b2f0-2301110d1b1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs"
Nov 03 20:46:24.590992 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.590973    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f02f361-0848-4bda-b2f0-2301110d1b1f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs\" (UID: \"8f02f361-0848-4bda-b2f0-2301110d1b1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs"
Nov 03 20:46:24.591119 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.591093    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f02f361-0848-4bda-b2f0-2301110d1b1f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs\" (UID: \"8f02f361-0848-4bda-b2f0-2301110d1b1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs"
Nov 03 20:46:24.616305 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.616274    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2"]
Nov 03 20:46:24.619996 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.619974    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2"
Nov 03 20:46:24.631673 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.631647    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2"]
Nov 03 20:46:24.691735 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.691695    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f02f361-0848-4bda-b2f0-2301110d1b1f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs\" (UID: \"8f02f361-0848-4bda-b2f0-2301110d1b1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs"
Nov 03 20:46:24.691931 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.691746    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67355315-4e2e-40b6-ac13-a3430749327e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9\" (UID: \"67355315-4e2e-40b6-ac13-a3430749327e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9"
Nov 03 20:46:24.691931 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.691778    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/287635c7-68c8-4261-9d6a-4c9f5ef20bd4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2\" (UID: \"287635c7-68c8-4261-9d6a-4c9f5ef20bd4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2"
Nov 03 20:46:24.691931 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.691850    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnjrh\" (UniqueName: \"kubernetes.io/projected/67355315-4e2e-40b6-ac13-a3430749327e-kube-api-access-wnjrh\") pod \"payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9\" (UID: \"67355315-4e2e-40b6-ac13-a3430749327e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9"
Nov 03 20:46:24.692058 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.691992    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t74ph\" (UniqueName: \"kubernetes.io/projected/287635c7-68c8-4261-9d6a-4c9f5ef20bd4-kube-api-access-t74ph\") pod \"payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2\" (UID: \"287635c7-68c8-4261-9d6a-4c9f5ef20bd4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2"
Nov 03 20:46:24.692058 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.692041    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f02f361-0848-4bda-b2f0-2301110d1b1f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs\" (UID: \"8f02f361-0848-4bda-b2f0-2301110d1b1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs"
Nov 03 20:46:24.692154 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.692134    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f02f361-0848-4bda-b2f0-2301110d1b1f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs\" (UID: \"8f02f361-0848-4bda-b2f0-2301110d1b1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs"
Nov 03 20:46:24.692192 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.692177    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75c9h\" (UniqueName: \"kubernetes.io/projected/8f02f361-0848-4bda-b2f0-2301110d1b1f-kube-api-access-75c9h\") pod \"maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs\" (UID: \"8f02f361-0848-4bda-b2f0-2301110d1b1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs"
Nov 03 20:46:24.692857 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.692824    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f02f361-0848-4bda-b2f0-2301110d1b1f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs\" (UID: \"8f02f361-0848-4bda-b2f0-2301110d1b1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs"
Nov 03 20:46:24.694627 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.694604    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f02f361-0848-4bda-b2f0-2301110d1b1f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs\" (UID: \"8f02f361-0848-4bda-b2f0-2301110d1b1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs"
Nov 03 20:46:24.694751 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.694683    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f02f361-0848-4bda-b2f0-2301110d1b1f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs\" (UID: \"8f02f361-0848-4bda-b2f0-2301110d1b1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs"
Nov 03 20:46:24.706645 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.706602    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75c9h\" (UniqueName: \"kubernetes.io/projected/8f02f361-0848-4bda-b2f0-2301110d1b1f-kube-api-access-75c9h\") pod \"maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs\" (UID: \"8f02f361-0848-4bda-b2f0-2301110d1b1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs"
Nov 03 20:46:24.788552 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.788508    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs"
Nov 03 20:46:24.793435 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.793390    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67355315-4e2e-40b6-ac13-a3430749327e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9\" (UID: \"67355315-4e2e-40b6-ac13-a3430749327e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9"
Nov 03 20:46:24.793577 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.793449    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/287635c7-68c8-4261-9d6a-4c9f5ef20bd4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2\" (UID: \"287635c7-68c8-4261-9d6a-4c9f5ef20bd4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2"
Nov 03 20:46:24.793577 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.793482    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnjrh\" (UniqueName: \"kubernetes.io/projected/67355315-4e2e-40b6-ac13-a3430749327e-kube-api-access-wnjrh\") pod \"payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9\" (UID: \"67355315-4e2e-40b6-ac13-a3430749327e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9"
Nov 03 20:46:24.793577 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.793526    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t74ph\" (UniqueName: \"kubernetes.io/projected/287635c7-68c8-4261-9d6a-4c9f5ef20bd4-kube-api-access-t74ph\") pod \"payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2\" (UID: \"287635c7-68c8-4261-9d6a-4c9f5ef20bd4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2"
Nov 03 20:46:24.794296 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.794276    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/287635c7-68c8-4261-9d6a-4c9f5ef20bd4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2\" (UID: \"287635c7-68c8-4261-9d6a-4c9f5ef20bd4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2"
Nov 03 20:46:24.794385 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.794294    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67355315-4e2e-40b6-ac13-a3430749327e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9\" (UID: \"67355315-4e2e-40b6-ac13-a3430749327e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9"
Nov 03 20:46:24.806057 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.806031    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnjrh\" (UniqueName: \"kubernetes.io/projected/67355315-4e2e-40b6-ac13-a3430749327e-kube-api-access-wnjrh\") pod \"payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9\" (UID: \"67355315-4e2e-40b6-ac13-a3430749327e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9"
Nov 03 20:46:24.806232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.806095    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t74ph\" (UniqueName: \"kubernetes.io/projected/287635c7-68c8-4261-9d6a-4c9f5ef20bd4-kube-api-access-t74ph\") pod \"payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2\" (UID: \"287635c7-68c8-4261-9d6a-4c9f5ef20bd4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2"
Nov 03 20:46:24.893646 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.893605    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9"
Nov 03 20:46:24.934298 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:24.934254    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2"
Nov 03 20:46:25.079970 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:25.079877    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs"]
Nov 03 20:46:25.085310 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:46:25.085275    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f02f361_0848_4bda_b2f0_2301110d1b1f.slice/crio-f55e29b17054130eb4b59268bccd7409745cc842504316be56f4bccdd032842f WatchSource:0}: Error finding container f55e29b17054130eb4b59268bccd7409745cc842504316be56f4bccdd032842f: Status 404 returned error can't find the container with id f55e29b17054130eb4b59268bccd7409745cc842504316be56f4bccdd032842f
Nov 03 20:46:25.087340 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:25.087312    5591 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 20:46:25.095609 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:25.095574    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs" event={"ID":"8f02f361-0848-4bda-b2f0-2301110d1b1f","Type":"ContainerStarted","Data":"f55e29b17054130eb4b59268bccd7409745cc842504316be56f4bccdd032842f"}
Nov 03 20:46:25.176391 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:25.176359    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9"]
Nov 03 20:46:25.178407 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:46:25.178376    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67355315_4e2e_40b6_ac13_a3430749327e.slice/crio-4c1dec30c2ac81239f398d5aee5502b7ed188c8a40a04f24e81e4e85c88a8288 WatchSource:0}: Error finding container 4c1dec30c2ac81239f398d5aee5502b7ed188c8a40a04f24e81e4e85c88a8288: Status 404 returned error can't find the container with id 4c1dec30c2ac81239f398d5aee5502b7ed188c8a40a04f24e81e4e85c88a8288
Nov 03 20:46:25.230559 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:25.230528    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2"]
Nov 03 20:46:25.233022 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:46:25.232988    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod287635c7_68c8_4261_9d6a_4c9f5ef20bd4.slice/crio-3655a0ce0767027936105f0556563a90731a9e1a05f3f3e7021d2a27c6db6b9a WatchSource:0}: Error finding container 3655a0ce0767027936105f0556563a90731a9e1a05f3f3e7021d2a27c6db6b9a: Status 404 returned error can't find the container with id 3655a0ce0767027936105f0556563a90731a9e1a05f3f3e7021d2a27c6db6b9a
Nov 03 20:46:26.102176 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:26.102137    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9" event={"ID":"67355315-4e2e-40b6-ac13-a3430749327e","Type":"ContainerStarted","Data":"12ecf7ee7cbdc9784b06dfd650babc746478380365b3cb575eb96880334ee60b"}
Nov 03 20:46:26.102692 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:26.102231    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9" event={"ID":"67355315-4e2e-40b6-ac13-a3430749327e","Type":"ContainerStarted","Data":"4c1dec30c2ac81239f398d5aee5502b7ed188c8a40a04f24e81e4e85c88a8288"}
Nov 03 20:46:26.102692 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:26.102383    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9"
Nov 03 20:46:26.104065 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:26.104038    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs" event={"ID":"8f02f361-0848-4bda-b2f0-2301110d1b1f","Type":"ContainerStarted","Data":"ac633fd2fc0731123a38fc644ed2efb111de29f761e202e590eace05f4cade80"}
Nov 03 20:46:26.104275 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:26.104097    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs"
Nov 03 20:46:26.105966 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:26.105942    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2" event={"ID":"287635c7-68c8-4261-9d6a-4c9f5ef20bd4","Type":"ContainerStarted","Data":"29d22f49f800bb63e7e2e0423fad7ec4f9632fdb8b91e25396e122c6681c58e9"}
Nov 03 20:46:26.106070 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:26.105982    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2" event={"ID":"287635c7-68c8-4261-9d6a-4c9f5ef20bd4","Type":"ContainerStarted","Data":"3655a0ce0767027936105f0556563a90731a9e1a05f3f3e7021d2a27c6db6b9a"}
Nov 03 20:46:26.106070 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:26.106024    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2"
Nov 03 20:46:26.119620 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:26.119566    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9" podStartSLOduration=2.11954904 podStartE2EDuration="2.11954904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:26.118982875 +0000 UTC m=+4134.891275897" watchObservedRunningTime="2026-11-03 20:46:26.11954904 +0000 UTC m=+4134.891842058"
Nov 03 20:46:26.140767 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:26.140701    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2" podStartSLOduration=2.140686544 podStartE2EDuration="2.140686544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:26.138925374 +0000 UTC m=+4134.911218392" watchObservedRunningTime="2026-11-03 20:46:26.140686544 +0000 UTC m=+4134.912979562"
Nov 03 20:46:26.160402 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:26.160345    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs" podStartSLOduration=1.594796981 podStartE2EDuration="2.160329997s" totalImagesPullingTime="565.533016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:26.15817546 +0000 UTC m=+4134.930468480" watchObservedRunningTime="2026-11-03 20:46:26.160329997 +0000 UTC m=+4134.932623015"
Nov 03 20:46:26.513653 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:26.513627    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:46:28.726973 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:46:28.726931    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:46:32.117549 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:32.117519    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs"
Nov 03 20:46:37.115282 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:37.115250    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9"
Nov 03 20:46:37.115829 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:37.115321    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2"
Nov 03 20:46:37.178593 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:37.178510    5591 generic.go:356] "Generic (PLEG): container finished" podID="1b577df4-3ef8-4c04-8f9e-e454254267d6" containerID="95554c60f1c41cbc6106ca2be125586771e9ba47098c88d2bbb7cf4273ed8d31" exitCode=0
Nov 03 20:46:37.178593 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:37.178580    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk" event={"ID":"1b577df4-3ef8-4c04-8f9e-e454254267d6","Type":"ContainerDied","Data":"95554c60f1c41cbc6106ca2be125586771e9ba47098c88d2bbb7cf4273ed8d31"}
Nov 03 20:46:37.178999 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:37.178978    5591 scope.go:122] "RemoveContainer" containerID="95554c60f1c41cbc6106ca2be125586771e9ba47098c88d2bbb7cf4273ed8d31"
Nov 03 20:46:37.180393 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:37.180352    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:46:37.180509 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:37.180442    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:46:37.180509 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:37.180478    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:46:37.333759 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:37.333726    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:46:38.205952 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:38.205912    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk" event={"ID":"1b577df4-3ef8-4c04-8f9e-e454254267d6","Type":"ContainerStarted","Data":"6686b7e6d02dca1a1e7e3cb48396a9dc56e34aa8a1e0114e50746f948d827906"}
Nov 03 20:46:38.545581 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:38.545541    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:38.546993 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:38.546964    5591 patch_prober.go:29] interesting pod/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" start-of-body=
Nov 03 20:46:38.547154 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:38.547020    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk" podUID="1b577df4-3ef8-4c04-8f9e-e454254267d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused"
Nov 03 20:46:39.546086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:39.546045    5591 patch_prober.go:29] interesting pod/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" start-of-body=
Nov 03 20:46:39.546564 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:39.546113    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk" podUID="1b577df4-3ef8-4c04-8f9e-e454254267d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused"
Nov 03 20:46:40.545420 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:40.545378    5591 patch_prober.go:29] interesting pod/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" start-of-body=
Nov 03 20:46:40.545617 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:40.545440    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk" podUID="1b577df4-3ef8-4c04-8f9e-e454254267d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused"
Nov 03 20:46:41.545529 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:41.545441    5591 patch_prober.go:29] interesting pod/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" start-of-body=
Nov 03 20:46:41.545529 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:41.545506    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk" podUID="1b577df4-3ef8-4c04-8f9e-e454254267d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused"
Nov 03 20:46:42.025762 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:42.025734    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:42.545789 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:42.545746    5591 patch_prober.go:29] interesting pod/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" start-of-body=
Nov 03 20:46:42.546323 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:42.545814    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk" podUID="1b577df4-3ef8-4c04-8f9e-e454254267d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused"
Nov 03 20:46:43.739397 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:43.739365    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:43.740862 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:43.740837    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:46:52.709742 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:46:52.709649    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:46:58.723378 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:46:58.723329    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:47:16.495433 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:16.495393    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:47:23.134524 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:23.134488    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2"]
Nov 03 20:47:23.154731 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:23.154696    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9"]
Nov 03 20:47:24.442681 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:24.442640    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs"]
Nov 03 20:47:24.443189 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:24.442891    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs" podUID="8f02f361-0848-4bda-b2f0-2301110d1b1f" containerName="maas-api" containerID="cri-o://ac633fd2fc0731123a38fc644ed2efb111de29f761e202e590eace05f4cade80" gracePeriod=30
Nov 03 20:47:24.900243 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:24.900219    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs"
Nov 03 20:47:24.949756 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:24.949723    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f02f361-0848-4bda-b2f0-2301110d1b1f-kube-api-access-75c9h\" (UniqueName: \"kubernetes.io/projected/8f02f361-0848-4bda-b2f0-2301110d1b1f-kube-api-access-75c9h\") pod \"8f02f361-0848-4bda-b2f0-2301110d1b1f\" (UID: \"8f02f361-0848-4bda-b2f0-2301110d1b1f\") "
Nov 03 20:47:24.949978 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:24.949781    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f02f361-0848-4bda-b2f0-2301110d1b1f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f02f361-0848-4bda-b2f0-2301110d1b1f-metrics-certs\") pod \"8f02f361-0848-4bda-b2f0-2301110d1b1f\" (UID: \"8f02f361-0848-4bda-b2f0-2301110d1b1f\") "
Nov 03 20:47:24.949978 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:24.949819    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f02f361-0848-4bda-b2f0-2301110d1b1f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f02f361-0848-4bda-b2f0-2301110d1b1f-maas-api-tls\") pod \"8f02f361-0848-4bda-b2f0-2301110d1b1f\" (UID: \"8f02f361-0848-4bda-b2f0-2301110d1b1f\") "
Nov 03 20:47:24.949978 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:24.949930    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f02f361-0848-4bda-b2f0-2301110d1b1f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f02f361-0848-4bda-b2f0-2301110d1b1f-openshift-service-ca\") pod \"8f02f361-0848-4bda-b2f0-2301110d1b1f\" (UID: \"8f02f361-0848-4bda-b2f0-2301110d1b1f\") "
Nov 03 20:47:24.950331 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:24.950305    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f02f361-0848-4bda-b2f0-2301110d1b1f-openshift-service-ca" pod "8f02f361-0848-4bda-b2f0-2301110d1b1f" (UID: "8f02f361-0848-4bda-b2f0-2301110d1b1f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:47:24.950517 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:24.950494    5591 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f02f361-0848-4bda-b2f0-2301110d1b1f-openshift-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:24.952103 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:24.952078    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f02f361-0848-4bda-b2f0-2301110d1b1f-metrics-certs" pod "8f02f361-0848-4bda-b2f0-2301110d1b1f" (UID: "8f02f361-0848-4bda-b2f0-2301110d1b1f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:47:24.952617 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:24.952599    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f02f361-0848-4bda-b2f0-2301110d1b1f-maas-api-tls" pod "8f02f361-0848-4bda-b2f0-2301110d1b1f" (UID: "8f02f361-0848-4bda-b2f0-2301110d1b1f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:47:24.952711 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:24.952643    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f02f361-0848-4bda-b2f0-2301110d1b1f-kube-api-access-75c9h" pod "8f02f361-0848-4bda-b2f0-2301110d1b1f" (UID: "8f02f361-0848-4bda-b2f0-2301110d1b1f"). InnerVolumeSpecName "kube-api-access-75c9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:47:25.051176 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:25.051135    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75c9h\" (UniqueName: \"kubernetes.io/projected/8f02f361-0848-4bda-b2f0-2301110d1b1f-kube-api-access-75c9h\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:25.051176 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:25.051170    5591 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f02f361-0848-4bda-b2f0-2301110d1b1f-metrics-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:25.051176 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:25.051181    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f02f361-0848-4bda-b2f0-2301110d1b1f-maas-api-tls\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:25.459214 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:25.459177    5591 generic.go:356] "Generic (PLEG): container finished" podID="8f02f361-0848-4bda-b2f0-2301110d1b1f" containerID="ac633fd2fc0731123a38fc644ed2efb111de29f761e202e590eace05f4cade80" exitCode=0
Nov 03 20:47:25.459661 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:25.459252    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs"
Nov 03 20:47:25.459661 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:25.459264    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs" event={"ID":"8f02f361-0848-4bda-b2f0-2301110d1b1f","Type":"ContainerDied","Data":"ac633fd2fc0731123a38fc644ed2efb111de29f761e202e590eace05f4cade80"}
Nov 03 20:47:25.459661 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:25.459319    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs" event={"ID":"8f02f361-0848-4bda-b2f0-2301110d1b1f","Type":"ContainerDied","Data":"f55e29b17054130eb4b59268bccd7409745cc842504316be56f4bccdd032842f"}
Nov 03 20:47:25.459661 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:25.459340    5591 scope.go:122] "RemoveContainer" containerID="ac633fd2fc0731123a38fc644ed2efb111de29f761e202e590eace05f4cade80"
Nov 03 20:47:25.469153 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:25.469132    5591 scope.go:122] "RemoveContainer" containerID="ac633fd2fc0731123a38fc644ed2efb111de29f761e202e590eace05f4cade80"
Nov 03 20:47:25.469456 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:47:25.469435    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac633fd2fc0731123a38fc644ed2efb111de29f761e202e590eace05f4cade80\": container with ID starting with ac633fd2fc0731123a38fc644ed2efb111de29f761e202e590eace05f4cade80 not found: ID does not exist" containerID="ac633fd2fc0731123a38fc644ed2efb111de29f761e202e590eace05f4cade80"
Nov 03 20:47:25.469508 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:25.469467    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac633fd2fc0731123a38fc644ed2efb111de29f761e202e590eace05f4cade80"} err="failed to get container status \"ac633fd2fc0731123a38fc644ed2efb111de29f761e202e590eace05f4cade80\": rpc error: code = NotFound desc = could not find container \"ac633fd2fc0731123a38fc644ed2efb111de29f761e202e590eace05f4cade80\": container with ID starting with ac633fd2fc0731123a38fc644ed2efb111de29f761e202e590eace05f4cade80 not found: ID does not exist"
Nov 03 20:47:25.477760 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:25.477731    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs"]
Nov 03 20:47:25.484596 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:25.484571    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-e7bf5b-6d98cb6888-pjxjs"]
Nov 03 20:47:26.116038 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:26.116001    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:47:27.348246 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:27.348185    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f02f361-0848-4bda-b2f0-2301110d1b1f" path="/var/lib/kubelet/pods/8f02f361-0848-4bda-b2f0-2301110d1b1f/volumes"
Nov 03 20:47:28.134851 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.134806    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2" podUID="287635c7-68c8-4261-9d6a-4c9f5ef20bd4" containerName="payload-processing" containerID="cri-o://29d22f49f800bb63e7e2e0423fad7ec4f9632fdb8b91e25396e122c6681c58e9" gracePeriod=25
Nov 03 20:47:28.155386 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.155332    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9" podUID="67355315-4e2e-40b6-ac13-a3430749327e" containerName="payload-pre-processing" containerID="cri-o://12ecf7ee7cbdc9784b06dfd650babc746478380365b3cb575eb96880334ee60b" gracePeriod=25
Nov 03 20:47:28.414912 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.414891    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2"
Nov 03 20:47:28.448419 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.448397    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9"
Nov 03 20:47:28.477948 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.477906    5591 generic.go:356] "Generic (PLEG): container finished" podID="67355315-4e2e-40b6-ac13-a3430749327e" containerID="12ecf7ee7cbdc9784b06dfd650babc746478380365b3cb575eb96880334ee60b" exitCode=0
Nov 03 20:47:28.477948 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.477949    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9"
Nov 03 20:47:28.478196 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.477982    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9" event={"ID":"67355315-4e2e-40b6-ac13-a3430749327e","Type":"ContainerDied","Data":"12ecf7ee7cbdc9784b06dfd650babc746478380365b3cb575eb96880334ee60b"}
Nov 03 20:47:28.478196 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.478037    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9" event={"ID":"67355315-4e2e-40b6-ac13-a3430749327e","Type":"ContainerDied","Data":"4c1dec30c2ac81239f398d5aee5502b7ed188c8a40a04f24e81e4e85c88a8288"}
Nov 03 20:47:28.478196 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.478053    5591 scope.go:122] "RemoveContainer" containerID="12ecf7ee7cbdc9784b06dfd650babc746478380365b3cb575eb96880334ee60b"
Nov 03 20:47:28.479596 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.479571    5591 generic.go:356] "Generic (PLEG): container finished" podID="287635c7-68c8-4261-9d6a-4c9f5ef20bd4" containerID="29d22f49f800bb63e7e2e0423fad7ec4f9632fdb8b91e25396e122c6681c58e9" exitCode=0
Nov 03 20:47:28.479723 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.479609    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2" event={"ID":"287635c7-68c8-4261-9d6a-4c9f5ef20bd4","Type":"ContainerDied","Data":"29d22f49f800bb63e7e2e0423fad7ec4f9632fdb8b91e25396e122c6681c58e9"}
Nov 03 20:47:28.479723 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.479626    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2"
Nov 03 20:47:28.479723 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.479651    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2" event={"ID":"287635c7-68c8-4261-9d6a-4c9f5ef20bd4","Type":"ContainerDied","Data":"3655a0ce0767027936105f0556563a90731a9e1a05f3f3e7021d2a27c6db6b9a"}
Nov 03 20:47:28.485746 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.485725    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/287635c7-68c8-4261-9d6a-4c9f5ef20bd4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/287635c7-68c8-4261-9d6a-4c9f5ef20bd4-plugins-config-volume\") pod \"287635c7-68c8-4261-9d6a-4c9f5ef20bd4\" (UID: \"287635c7-68c8-4261-9d6a-4c9f5ef20bd4\") "
Nov 03 20:47:28.485879 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.485868    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/287635c7-68c8-4261-9d6a-4c9f5ef20bd4-kube-api-access-t74ph\" (UniqueName: \"kubernetes.io/projected/287635c7-68c8-4261-9d6a-4c9f5ef20bd4-kube-api-access-t74ph\") pod \"287635c7-68c8-4261-9d6a-4c9f5ef20bd4\" (UID: \"287635c7-68c8-4261-9d6a-4c9f5ef20bd4\") "
Nov 03 20:47:28.485947 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.485893    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67355315-4e2e-40b6-ac13-a3430749327e-kube-api-access-wnjrh\" (UniqueName: \"kubernetes.io/projected/67355315-4e2e-40b6-ac13-a3430749327e-kube-api-access-wnjrh\") pod \"67355315-4e2e-40b6-ac13-a3430749327e\" (UID: \"67355315-4e2e-40b6-ac13-a3430749327e\") "
Nov 03 20:47:28.485947 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.485937    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67355315-4e2e-40b6-ac13-a3430749327e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67355315-4e2e-40b6-ac13-a3430749327e-plugins-config-volume\") pod \"67355315-4e2e-40b6-ac13-a3430749327e\" (UID: \"67355315-4e2e-40b6-ac13-a3430749327e\") "
Nov 03 20:47:28.486168 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.486146    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/287635c7-68c8-4261-9d6a-4c9f5ef20bd4-plugins-config-volume" pod "287635c7-68c8-4261-9d6a-4c9f5ef20bd4" (UID: "287635c7-68c8-4261-9d6a-4c9f5ef20bd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:47:28.486421 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.486388    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67355315-4e2e-40b6-ac13-a3430749327e-plugins-config-volume" pod "67355315-4e2e-40b6-ac13-a3430749327e" (UID: "67355315-4e2e-40b6-ac13-a3430749327e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:47:28.486678 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.486410    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/287635c7-68c8-4261-9d6a-4c9f5ef20bd4-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:28.488113 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.488082    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/287635c7-68c8-4261-9d6a-4c9f5ef20bd4-kube-api-access-t74ph" pod "287635c7-68c8-4261-9d6a-4c9f5ef20bd4" (UID: "287635c7-68c8-4261-9d6a-4c9f5ef20bd4"). InnerVolumeSpecName "kube-api-access-t74ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:47:28.488416 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.488327    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67355315-4e2e-40b6-ac13-a3430749327e-kube-api-access-wnjrh" pod "67355315-4e2e-40b6-ac13-a3430749327e" (UID: "67355315-4e2e-40b6-ac13-a3430749327e"). InnerVolumeSpecName "kube-api-access-wnjrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:47:28.488505 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.488491    5591 scope.go:122] "RemoveContainer" containerID="12ecf7ee7cbdc9784b06dfd650babc746478380365b3cb575eb96880334ee60b"
Nov 03 20:47:28.488804 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:47:28.488773    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12ecf7ee7cbdc9784b06dfd650babc746478380365b3cb575eb96880334ee60b\": container with ID starting with 12ecf7ee7cbdc9784b06dfd650babc746478380365b3cb575eb96880334ee60b not found: ID does not exist" containerID="12ecf7ee7cbdc9784b06dfd650babc746478380365b3cb575eb96880334ee60b"
Nov 03 20:47:28.488863 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.488808    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12ecf7ee7cbdc9784b06dfd650babc746478380365b3cb575eb96880334ee60b"} err="failed to get container status \"12ecf7ee7cbdc9784b06dfd650babc746478380365b3cb575eb96880334ee60b\": rpc error: code = NotFound desc = could not find container \"12ecf7ee7cbdc9784b06dfd650babc746478380365b3cb575eb96880334ee60b\": container with ID starting with 12ecf7ee7cbdc9784b06dfd650babc746478380365b3cb575eb96880334ee60b not found: ID does not exist"
Nov 03 20:47:28.488863 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.488822    5591 scope.go:122] "RemoveContainer" containerID="29d22f49f800bb63e7e2e0423fad7ec4f9632fdb8b91e25396e122c6681c58e9"
Nov 03 20:47:28.497124 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.497097    5591 scope.go:122] "RemoveContainer" containerID="29d22f49f800bb63e7e2e0423fad7ec4f9632fdb8b91e25396e122c6681c58e9"
Nov 03 20:47:28.497417 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:47:28.497397    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29d22f49f800bb63e7e2e0423fad7ec4f9632fdb8b91e25396e122c6681c58e9\": container with ID starting with 29d22f49f800bb63e7e2e0423fad7ec4f9632fdb8b91e25396e122c6681c58e9 not found: ID does not exist" containerID="29d22f49f800bb63e7e2e0423fad7ec4f9632fdb8b91e25396e122c6681c58e9"
Nov 03 20:47:28.497513 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.497421    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29d22f49f800bb63e7e2e0423fad7ec4f9632fdb8b91e25396e122c6681c58e9"} err="failed to get container status \"29d22f49f800bb63e7e2e0423fad7ec4f9632fdb8b91e25396e122c6681c58e9\": rpc error: code = NotFound desc = could not find container \"29d22f49f800bb63e7e2e0423fad7ec4f9632fdb8b91e25396e122c6681c58e9\": container with ID starting with 29d22f49f800bb63e7e2e0423fad7ec4f9632fdb8b91e25396e122c6681c58e9 not found: ID does not exist"
Nov 03 20:47:28.588089 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.588059    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t74ph\" (UniqueName: \"kubernetes.io/projected/287635c7-68c8-4261-9d6a-4c9f5ef20bd4-kube-api-access-t74ph\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:28.588089 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.588088    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnjrh\" (UniqueName: \"kubernetes.io/projected/67355315-4e2e-40b6-ac13-a3430749327e-kube-api-access-wnjrh\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:28.588089 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.588098    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67355315-4e2e-40b6-ac13-a3430749327e-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:47:28.723044 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:47:28.722947    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:47:28.802160 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.802134    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9"]
Nov 03 20:47:28.806447 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.806418    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ww8r9"]
Nov 03 20:47:28.817315 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.817289    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2"]
Nov 03 20:47:28.821393 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:28.821371    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-e7bf5b-f8fcc8d74jm6r2"]
Nov 03 20:47:29.348120 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:29.348087    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="287635c7-68c8-4261-9d6a-4c9f5ef20bd4" path="/var/lib/kubelet/pods/287635c7-68c8-4261-9d6a-4c9f5ef20bd4/volumes"
Nov 03 20:47:29.348523 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:29.348509    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67355315-4e2e-40b6-ac13-a3430749327e" path="/var/lib/kubelet/pods/67355315-4e2e-40b6-ac13-a3430749327e/volumes"
Nov 03 20:47:34.183671 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:34.183638    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 20:47:34.191267 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:34.191246    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 20:47:38.072326 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:38.072293    5591 scope.go:122] "RemoveContainer" containerID="7494561d91fab4ed829766c3aa14f32b16e6fc9406d467fe5cb421b3f0e4d01a"
Nov 03 20:47:55.943254 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:55.943190    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd"]
Nov 03 20:47:55.943760 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:55.943696    5591 cpu_manager.go:414] "containerMap: removing container" podUID="67355315-4e2e-40b6-ac13-a3430749327e" containerName="payload-pre-processing"
Nov 03 20:47:55.943760 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:55.943711    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67355315-4e2e-40b6-ac13-a3430749327e" containerName="payload-pre-processing"
Nov 03 20:47:55.943760 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:55.943739    5591 cpu_manager.go:414] "containerMap: removing container" podUID="287635c7-68c8-4261-9d6a-4c9f5ef20bd4" containerName="payload-processing"
Nov 03 20:47:55.943760 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:55.943748    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="287635c7-68c8-4261-9d6a-4c9f5ef20bd4" containerName="payload-processing"
Nov 03 20:47:55.944039 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:55.943767    5591 cpu_manager.go:414] "containerMap: removing container" podUID="8f02f361-0848-4bda-b2f0-2301110d1b1f" containerName="maas-api"
Nov 03 20:47:55.944039 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:55.943773    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f02f361-0848-4bda-b2f0-2301110d1b1f" containerName="maas-api"
Nov 03 20:47:55.944039 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:55.943851    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="67355315-4e2e-40b6-ac13-a3430749327e" containerName="payload-pre-processing"
Nov 03 20:47:55.944039 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:55.943863    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="287635c7-68c8-4261-9d6a-4c9f5ef20bd4" containerName="payload-processing"
Nov 03 20:47:55.944039 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:55.943873    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f02f361-0848-4bda-b2f0-2301110d1b1f" containerName="maas-api"
Nov 03 20:47:55.946974 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:55.946953    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd"
Nov 03 20:47:55.957150 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:55.957128    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd"]
Nov 03 20:47:56.023744 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:56.023715    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-e7bf5b\""
Nov 03 20:47:56.027928 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:56.027908    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-4db0eb42-dockercfg-s6fj8\""
Nov 03 20:47:56.054837 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:56.054793    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7932c083-3119-49be-a514-09dcb51436c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd\" (UID: \"7932c083-3119-49be-a514-09dcb51436c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd"
Nov 03 20:47:56.055030 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:56.054965    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sgql\" (UniqueName: \"kubernetes.io/projected/7932c083-3119-49be-a514-09dcb51436c3-kube-api-access-8sgql\") pod \"payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd\" (UID: \"7932c083-3119-49be-a514-09dcb51436c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd"
Nov 03 20:47:56.156818 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:56.156780    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7932c083-3119-49be-a514-09dcb51436c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd\" (UID: \"7932c083-3119-49be-a514-09dcb51436c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd"
Nov 03 20:47:56.157026 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:56.156898    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sgql\" (UniqueName: \"kubernetes.io/projected/7932c083-3119-49be-a514-09dcb51436c3-kube-api-access-8sgql\") pod \"payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd\" (UID: \"7932c083-3119-49be-a514-09dcb51436c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd"
Nov 03 20:47:56.157550 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:56.157527    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7932c083-3119-49be-a514-09dcb51436c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd\" (UID: \"7932c083-3119-49be-a514-09dcb51436c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd"
Nov 03 20:47:56.171180 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:56.171156    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sgql\" (UniqueName: \"kubernetes.io/projected/7932c083-3119-49be-a514-09dcb51436c3-kube-api-access-8sgql\") pod \"payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd\" (UID: \"7932c083-3119-49be-a514-09dcb51436c3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd"
Nov 03 20:47:56.266463 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:56.266424    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd"
Nov 03 20:47:56.546122 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:56.546095    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd"]
Nov 03 20:47:56.548603 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:47:56.548573    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7932c083_3119_49be_a514_09dcb51436c3.slice/crio-e1bc9fbb4170ad6375fa23e59b94681356bd95c32dfe77768c1b4e9c3edca0fc WatchSource:0}: Error finding container e1bc9fbb4170ad6375fa23e59b94681356bd95c32dfe77768c1b4e9c3edca0fc: Status 404 returned error can't find the container with id e1bc9fbb4170ad6375fa23e59b94681356bd95c32dfe77768c1b4e9c3edca0fc
Nov 03 20:47:56.629533 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:56.629495    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd" event={"ID":"7932c083-3119-49be-a514-09dcb51436c3","Type":"ContainerStarted","Data":"a95256ab514553664857713d4fe84a067ecccc7f67c70af6ec8eb6a65e686bfb"}
Nov 03 20:47:56.629666 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:56.629563    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd" event={"ID":"7932c083-3119-49be-a514-09dcb51436c3","Type":"ContainerStarted","Data":"e1bc9fbb4170ad6375fa23e59b94681356bd95c32dfe77768c1b4e9c3edca0fc"}
Nov 03 20:47:56.629738 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:56.629718    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd"
Nov 03 20:47:56.647762 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:56.647703    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd" podStartSLOduration=1.647687972 podStartE2EDuration="1.647687972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:56.64766639 +0000 UTC m=+4225.419959410" watchObservedRunningTime="2026-11-03 20:47:56.647687972 +0000 UTC m=+4225.419980988"
Nov 03 20:47:58.148591 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:47:58.148560    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:47:58.724444 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:47:58.724406    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:48:02.912915 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:02.912875    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:48:04.149380 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:04.149340    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95"]
Nov 03 20:48:04.153262 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:04.153241    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95"
Nov 03 20:48:04.160819 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:04.159317    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:48:04.160819 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:04.159651    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:48:04.160819 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:04.160601    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 20:48:04.160819 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:04.160652    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:48:04.163243 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:04.163143    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95"]
Nov 03 20:48:04.242282 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:04.242246    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3584490d-b324-4136-953e-16b969d6ab67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95\" (UID: \"3584490d-b324-4136-953e-16b969d6ab67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95"
Nov 03 20:48:04.242481 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:04.242380    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmndm\" (UniqueName: \"kubernetes.io/projected/3584490d-b324-4136-953e-16b969d6ab67-kube-api-access-bmndm\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95\" (UID: \"3584490d-b324-4136-953e-16b969d6ab67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95"
Nov 03 20:48:04.242481 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:04.242402    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3584490d-b324-4136-953e-16b969d6ab67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95\" (UID: \"3584490d-b324-4136-953e-16b969d6ab67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95"
Nov 03 20:48:04.343233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:04.343161    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmndm\" (UniqueName: \"kubernetes.io/projected/3584490d-b324-4136-953e-16b969d6ab67-kube-api-access-bmndm\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95\" (UID: \"3584490d-b324-4136-953e-16b969d6ab67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95"
Nov 03 20:48:04.343233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:04.343234    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3584490d-b324-4136-953e-16b969d6ab67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95\" (UID: \"3584490d-b324-4136-953e-16b969d6ab67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95"
Nov 03 20:48:04.343515 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:04.343315    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3584490d-b324-4136-953e-16b969d6ab67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95\" (UID: \"3584490d-b324-4136-953e-16b969d6ab67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95"
Nov 03 20:48:04.344047 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:04.344023    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3584490d-b324-4136-953e-16b969d6ab67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95\" (UID: \"3584490d-b324-4136-953e-16b969d6ab67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95"
Nov 03 20:48:04.346107 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:04.346086    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3584490d-b324-4136-953e-16b969d6ab67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95\" (UID: \"3584490d-b324-4136-953e-16b969d6ab67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95"
Nov 03 20:48:04.355539 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:04.355505    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmndm\" (UniqueName: \"kubernetes.io/projected/3584490d-b324-4136-953e-16b969d6ab67-kube-api-access-bmndm\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95\" (UID: \"3584490d-b324-4136-953e-16b969d6ab67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95"
Nov 03 20:48:04.468917 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:04.468829    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95"
Nov 03 20:48:04.749965 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:04.749937    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95"]
Nov 03 20:48:04.753156 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:48:04.753126    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3584490d_b324_4136_953e_16b969d6ab67.slice/crio-49371e06cec8b646c0d87609dbefbdb0814ed9bbee67f65206f1160cde6f0526 WatchSource:0}: Error finding container 49371e06cec8b646c0d87609dbefbdb0814ed9bbee67f65206f1160cde6f0526: Status 404 returned error can't find the container with id 49371e06cec8b646c0d87609dbefbdb0814ed9bbee67f65206f1160cde6f0526
Nov 03 20:48:05.680020 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:05.679971    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95" event={"ID":"3584490d-b324-4136-953e-16b969d6ab67","Type":"ContainerStarted","Data":"348a2dcc9f8df8047a68bed3763387cb3e9b9b2c1a5a6cf0b7ceb71e8ba9e535"}
Nov 03 20:48:05.680611 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:05.680036    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95" event={"ID":"3584490d-b324-4136-953e-16b969d6ab67","Type":"ContainerStarted","Data":"49371e06cec8b646c0d87609dbefbdb0814ed9bbee67f65206f1160cde6f0526"}
Nov 03 20:48:05.698656 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:05.698596    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95" podStartSLOduration=1.698580486 podStartE2EDuration="1.698580486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:05.697333656 +0000 UTC m=+4234.469626676" watchObservedRunningTime="2026-11-03 20:48:05.698580486 +0000 UTC m=+4234.470873540"
Nov 03 20:48:06.686870 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:06.686830    5591 generic.go:356] "Generic (PLEG): container finished" podID="3584490d-b324-4136-953e-16b969d6ab67" containerID="348a2dcc9f8df8047a68bed3763387cb3e9b9b2c1a5a6cf0b7ceb71e8ba9e535" exitCode=0
Nov 03 20:48:06.687333 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:06.686909    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95" event={"ID":"3584490d-b324-4136-953e-16b969d6ab67","Type":"ContainerDied","Data":"348a2dcc9f8df8047a68bed3763387cb3e9b9b2c1a5a6cf0b7ceb71e8ba9e535"}
Nov 03 20:48:07.638910 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:07.638876    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd"
Nov 03 20:48:07.845301 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:07.845272    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95"
Nov 03 20:48:07.879061 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:07.879030    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3584490d-b324-4136-953e-16b969d6ab67-kube-api-access-bmndm\" (UniqueName: \"kubernetes.io/projected/3584490d-b324-4136-953e-16b969d6ab67-kube-api-access-bmndm\") pod \"3584490d-b324-4136-953e-16b969d6ab67\" (UID: \"3584490d-b324-4136-953e-16b969d6ab67\") "
Nov 03 20:48:07.879244 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:07.879092    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3584490d-b324-4136-953e-16b969d6ab67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3584490d-b324-4136-953e-16b969d6ab67-maas-api-openshift-ca\") pod \"3584490d-b324-4136-953e-16b969d6ab67\" (UID: \"3584490d-b324-4136-953e-16b969d6ab67\") "
Nov 03 20:48:07.879244 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:07.879150    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3584490d-b324-4136-953e-16b969d6ab67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3584490d-b324-4136-953e-16b969d6ab67-maas-api-xks-ca\") pod \"3584490d-b324-4136-953e-16b969d6ab67\" (UID: \"3584490d-b324-4136-953e-16b969d6ab67\") "
Nov 03 20:48:07.879520 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:07.879481    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3584490d-b324-4136-953e-16b969d6ab67-maas-api-openshift-ca" pod "3584490d-b324-4136-953e-16b969d6ab67" (UID: "3584490d-b324-4136-953e-16b969d6ab67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:48:07.881446 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:07.881409    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3584490d-b324-4136-953e-16b969d6ab67-maas-api-xks-ca" pod "3584490d-b324-4136-953e-16b969d6ab67" (UID: "3584490d-b324-4136-953e-16b969d6ab67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:48:07.881446 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:07.881428    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3584490d-b324-4136-953e-16b969d6ab67-kube-api-access-bmndm" pod "3584490d-b324-4136-953e-16b969d6ab67" (UID: "3584490d-b324-4136-953e-16b969d6ab67"). InnerVolumeSpecName "kube-api-access-bmndm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:48:07.980709 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:07.980620    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmndm\" (UniqueName: \"kubernetes.io/projected/3584490d-b324-4136-953e-16b969d6ab67-kube-api-access-bmndm\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:07.980709 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:07.980653    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3584490d-b324-4136-953e-16b969d6ab67-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:07.980709 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:07.980664    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3584490d-b324-4136-953e-16b969d6ab67-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:08.707366 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:08.707328    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95"
Nov 03 20:48:08.707556 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:08.707358    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95" event={"ID":"3584490d-b324-4136-953e-16b969d6ab67","Type":"ContainerDied","Data":"49371e06cec8b646c0d87609dbefbdb0814ed9bbee67f65206f1160cde6f0526"}
Nov 03 20:48:08.707556 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:08.707432    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49371e06cec8b646c0d87609dbefbdb0814ed9bbee67f65206f1160cde6f0526"
Nov 03 20:48:14.400518 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:14.400424    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd"]
Nov 03 20:48:15.201274 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:15.201236    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95"]
Nov 03 20:48:15.204864 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:15.204831    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e7bf5b-gmz95"]
Nov 03 20:48:15.349539 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:15.349504    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3584490d-b324-4136-953e-16b969d6ab67" path="/var/lib/kubelet/pods/3584490d-b324-4136-953e-16b969d6ab67/volumes"
Nov 03 20:48:15.560171 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:15.560139    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"]
Nov 03 20:48:15.560667 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:15.560372    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk" podUID="1b577df4-3ef8-4c04-8f9e-e454254267d6" containerName="istio-proxy" containerID="cri-o://6686b7e6d02dca1a1e7e3cb48396a9dc56e34aa8a1e0114e50746f948d827906" gracePeriod=30
Nov 03 20:48:17.021678 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:17.021639    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:48:19.401437 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:19.401397    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd" podUID="7932c083-3119-49be-a514-09dcb51436c3" containerName="payload-pre-processing" containerID="cri-o://a95256ab514553664857713d4fe84a067ecccc7f67c70af6ec8eb6a65e686bfb" gracePeriod=25
Nov 03 20:48:19.665464 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:19.665395    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd"
Nov 03 20:48:19.771122 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:19.771084    5591 generic.go:356] "Generic (PLEG): container finished" podID="7932c083-3119-49be-a514-09dcb51436c3" containerID="a95256ab514553664857713d4fe84a067ecccc7f67c70af6ec8eb6a65e686bfb" exitCode=0
Nov 03 20:48:19.771364 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:19.771130    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd"
Nov 03 20:48:19.771364 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:19.771160    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd" event={"ID":"7932c083-3119-49be-a514-09dcb51436c3","Type":"ContainerDied","Data":"a95256ab514553664857713d4fe84a067ecccc7f67c70af6ec8eb6a65e686bfb"}
Nov 03 20:48:19.771364 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:19.771228    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd" event={"ID":"7932c083-3119-49be-a514-09dcb51436c3","Type":"ContainerDied","Data":"e1bc9fbb4170ad6375fa23e59b94681356bd95c32dfe77768c1b4e9c3edca0fc"}
Nov 03 20:48:19.771364 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:19.771252    5591 scope.go:122] "RemoveContainer" containerID="a95256ab514553664857713d4fe84a067ecccc7f67c70af6ec8eb6a65e686bfb"
Nov 03 20:48:19.780471 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:19.780450    5591 scope.go:122] "RemoveContainer" containerID="a95256ab514553664857713d4fe84a067ecccc7f67c70af6ec8eb6a65e686bfb"
Nov 03 20:48:19.780713 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:48:19.780695    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a95256ab514553664857713d4fe84a067ecccc7f67c70af6ec8eb6a65e686bfb\": container with ID starting with a95256ab514553664857713d4fe84a067ecccc7f67c70af6ec8eb6a65e686bfb not found: ID does not exist" containerID="a95256ab514553664857713d4fe84a067ecccc7f67c70af6ec8eb6a65e686bfb"
Nov 03 20:48:19.780781 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:19.780722    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a95256ab514553664857713d4fe84a067ecccc7f67c70af6ec8eb6a65e686bfb"} err="failed to get container status \"a95256ab514553664857713d4fe84a067ecccc7f67c70af6ec8eb6a65e686bfb\": rpc error: code = NotFound desc = could not find container \"a95256ab514553664857713d4fe84a067ecccc7f67c70af6ec8eb6a65e686bfb\": container with ID starting with a95256ab514553664857713d4fe84a067ecccc7f67c70af6ec8eb6a65e686bfb not found: ID does not exist"
Nov 03 20:48:19.800231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:19.800191    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7932c083-3119-49be-a514-09dcb51436c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7932c083-3119-49be-a514-09dcb51436c3-plugins-config-volume\") pod \"7932c083-3119-49be-a514-09dcb51436c3\" (UID: \"7932c083-3119-49be-a514-09dcb51436c3\") "
Nov 03 20:48:19.800329 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:19.800320    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7932c083-3119-49be-a514-09dcb51436c3-kube-api-access-8sgql\" (UniqueName: \"kubernetes.io/projected/7932c083-3119-49be-a514-09dcb51436c3-kube-api-access-8sgql\") pod \"7932c083-3119-49be-a514-09dcb51436c3\" (UID: \"7932c083-3119-49be-a514-09dcb51436c3\") "
Nov 03 20:48:19.800529 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:19.800510    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7932c083-3119-49be-a514-09dcb51436c3-plugins-config-volume" pod "7932c083-3119-49be-a514-09dcb51436c3" (UID: "7932c083-3119-49be-a514-09dcb51436c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:48:19.800637 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:19.800624    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7932c083-3119-49be-a514-09dcb51436c3-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:19.802478 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:19.802460    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7932c083-3119-49be-a514-09dcb51436c3-kube-api-access-8sgql" pod "7932c083-3119-49be-a514-09dcb51436c3" (UID: "7932c083-3119-49be-a514-09dcb51436c3"). InnerVolumeSpecName "kube-api-access-8sgql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:48:19.901171 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:19.901136    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sgql\" (UniqueName: \"kubernetes.io/projected/7932c083-3119-49be-a514-09dcb51436c3-kube-api-access-8sgql\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:20.095639 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:20.095609    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd"]
Nov 03 20:48:20.100165 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:20.100138    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-e7bf5b-785c9ss6hd"]
Nov 03 20:48:20.778653 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:20.778611    5591 generic.go:356] "Generic (PLEG): container finished" podID="1b577df4-3ef8-4c04-8f9e-e454254267d6" containerID="6686b7e6d02dca1a1e7e3cb48396a9dc56e34aa8a1e0114e50746f948d827906" exitCode=0
Nov 03 20:48:20.779120 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:20.778687    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk" event={"ID":"1b577df4-3ef8-4c04-8f9e-e454254267d6","Type":"ContainerDied","Data":"6686b7e6d02dca1a1e7e3cb48396a9dc56e34aa8a1e0114e50746f948d827906"}
Nov 03 20:48:20.779120 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:20.778762    5591 scope.go:122] "RemoveContainer" containerID="95554c60f1c41cbc6106ca2be125586771e9ba47098c88d2bbb7cf4273ed8d31"
Nov 03 20:48:20.813873 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:20.813849    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:48:20.910547 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:20.910460    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b577df4-3ef8-4c04-8f9e-e454254267d6-kube-api-access-kjxz7\" (UniqueName: \"kubernetes.io/projected/1b577df4-3ef8-4c04-8f9e-e454254267d6-kube-api-access-kjxz7\") pod \"1b577df4-3ef8-4c04-8f9e-e454254267d6\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") "
Nov 03 20:48:20.910547 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:20.910507    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-podinfo\") pod \"1b577df4-3ef8-4c04-8f9e-e454254267d6\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") "
Nov 03 20:48:20.910547 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:20.910546    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-data\") pod \"1b577df4-3ef8-4c04-8f9e-e454254267d6\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") "
Nov 03 20:48:20.910860 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:20.910565    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-token\" (UniqueName: \"kubernetes.io/projected/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-token\") pod \"1b577df4-3ef8-4c04-8f9e-e454254267d6\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") "
Nov 03 20:48:20.910860 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:20.910636    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-workload-certs\") pod \"1b577df4-3ef8-4c04-8f9e-e454254267d6\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") "
Nov 03 20:48:20.910860 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:20.910673    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-workload-socket\") pod \"1b577df4-3ef8-4c04-8f9e-e454254267d6\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") "
Nov 03 20:48:20.910860 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:20.910706    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-credential-socket\") pod \"1b577df4-3ef8-4c04-8f9e-e454254267d6\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") "
Nov 03 20:48:20.910860 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:20.910735    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-envoy\") pod \"1b577df4-3ef8-4c04-8f9e-e454254267d6\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") "
Nov 03 20:48:20.910860 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:20.910768    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b577df4-3ef8-4c04-8f9e-e454254267d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b577df4-3ef8-4c04-8f9e-e454254267d6-istiod-ca-cert\") pod \"1b577df4-3ef8-4c04-8f9e-e454254267d6\" (UID: \"1b577df4-3ef8-4c04-8f9e-e454254267d6\") "
Nov 03 20:48:20.911191 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:20.910925    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-data" pod "1b577df4-3ef8-4c04-8f9e-e454254267d6" (UID: "1b577df4-3ef8-4c04-8f9e-e454254267d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:48:20.911191 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:20.911040    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-credential-socket" pod "1b577df4-3ef8-4c04-8f9e-e454254267d6" (UID: "1b577df4-3ef8-4c04-8f9e-e454254267d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:48:20.911191 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:20.911150    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-workload-socket" pod "1b577df4-3ef8-4c04-8f9e-e454254267d6" (UID: "1b577df4-3ef8-4c04-8f9e-e454254267d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:48:20.911405 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:20.911263    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b577df4-3ef8-4c04-8f9e-e454254267d6-istiod-ca-cert" pod "1b577df4-3ef8-4c04-8f9e-e454254267d6" (UID: "1b577df4-3ef8-4c04-8f9e-e454254267d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:48:20.911405 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:20.911333    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:20.911405 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:20.911357    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:20.911405 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:20.911373    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:20.911587 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:20.911448    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-workload-certs" pod "1b577df4-3ef8-4c04-8f9e-e454254267d6" (UID: "1b577df4-3ef8-4c04-8f9e-e454254267d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:48:20.912974 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:20.912953    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-token" pod "1b577df4-3ef8-4c04-8f9e-e454254267d6" (UID: "1b577df4-3ef8-4c04-8f9e-e454254267d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:48:20.913077 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:20.913061    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-envoy" pod "1b577df4-3ef8-4c04-8f9e-e454254267d6" (UID: "1b577df4-3ef8-4c04-8f9e-e454254267d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:48:20.913447 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:20.913421    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b577df4-3ef8-4c04-8f9e-e454254267d6-kube-api-access-kjxz7" pod "1b577df4-3ef8-4c04-8f9e-e454254267d6" (UID: "1b577df4-3ef8-4c04-8f9e-e454254267d6"). InnerVolumeSpecName "kube-api-access-kjxz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:48:20.913447 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:20.913421    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-podinfo" pod "1b577df4-3ef8-4c04-8f9e-e454254267d6" (UID: "1b577df4-3ef8-4c04-8f9e-e454254267d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:48:21.012021 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:21.011988    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:21.012021 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:21.012016    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:21.012021 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:21.012026    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b577df4-3ef8-4c04-8f9e-e454254267d6-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:21.012313 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:21.012036    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjxz7\" (UniqueName: \"kubernetes.io/projected/1b577df4-3ef8-4c04-8f9e-e454254267d6-kube-api-access-kjxz7\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:21.012313 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:21.012046    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:21.012313 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:21.012055    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b577df4-3ef8-4c04-8f9e-e454254267d6-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:48:21.028647 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:21.028613    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:48:21.348191 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:21.348158    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7932c083-3119-49be-a514-09dcb51436c3" path="/var/lib/kubelet/pods/7932c083-3119-49be-a514-09dcb51436c3/volumes"
Nov 03 20:48:21.788593 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:21.788549    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk" event={"ID":"1b577df4-3ef8-4c04-8f9e-e454254267d6","Type":"ContainerDied","Data":"06af1692fc8f20764de54d6091db91f31e590aa63924485347dbc52c824d76b1"}
Nov 03 20:48:21.788593 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:21.788583    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"
Nov 03 20:48:21.789295 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:21.788633    5591 scope.go:122] "RemoveContainer" containerID="6686b7e6d02dca1a1e7e3cb48396a9dc56e34aa8a1e0114e50746f948d827906"
Nov 03 20:48:21.808960 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:21.808927    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"]
Nov 03 20:48:21.815431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:21.815405    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e7bf5b-openshift-default-77fc5486dfjfjjk"]
Nov 03 20:48:23.353380 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:23.353334    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b577df4-3ef8-4c04-8f9e-e454254267d6" path="/var/lib/kubelet/pods/1b577df4-3ef8-4c04-8f9e-e454254267d6/volumes"
Nov 03 20:48:28.723493 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:48:28.723448    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:48:38.105460 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:38.105421    5591 scope.go:122] "RemoveContainer" containerID="ffb0c556089aee49a4605053b6976adab1144e2cdb93f2b183e0533619504763"
Nov 03 20:48:38.308026 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:38.307982    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:48:41.794235 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:41.792346    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-mrwmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:48:41.794235 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:48:41.792490    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt" podUID="e14ca8f3-c80b-4c0b-9564-ef9734b86de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:48:58.723059 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:48:58.723013    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:49:03.482606 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:03.482575    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cbx9w"]
Nov 03 20:49:03.483020 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:03.482995    5591 cpu_manager.go:414] "containerMap: removing container" podUID="3584490d-b324-4136-953e-16b969d6ab67" containerName="revoke-keys"
Nov 03 20:49:03.483020 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:03.483005    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3584490d-b324-4136-953e-16b969d6ab67" containerName="revoke-keys"
Nov 03 20:49:03.483020 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:03.483021    5591 cpu_manager.go:414] "containerMap: removing container" podUID="7932c083-3119-49be-a514-09dcb51436c3" containerName="payload-pre-processing"
Nov 03 20:49:03.483147 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:03.483027    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7932c083-3119-49be-a514-09dcb51436c3" containerName="payload-pre-processing"
Nov 03 20:49:03.483147 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:03.483038    5591 cpu_manager.go:414] "containerMap: removing container" podUID="1b577df4-3ef8-4c04-8f9e-e454254267d6" containerName="istio-proxy"
Nov 03 20:49:03.483147 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:03.483042    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b577df4-3ef8-4c04-8f9e-e454254267d6" containerName="istio-proxy"
Nov 03 20:49:03.483147 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:03.483112    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3584490d-b324-4136-953e-16b969d6ab67" containerName="revoke-keys"
Nov 03 20:49:03.483147 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:03.483128    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7932c083-3119-49be-a514-09dcb51436c3" containerName="payload-pre-processing"
Nov 03 20:49:03.483147 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:03.483138    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b577df4-3ef8-4c04-8f9e-e454254267d6" containerName="istio-proxy"
Nov 03 20:49:03.483391 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:03.483149    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b577df4-3ef8-4c04-8f9e-e454254267d6" containerName="istio-proxy"
Nov 03 20:49:03.486337 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:03.486318    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cbx9w"
Nov 03 20:49:03.489864 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:03.489843    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:49:03.490068 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:03.489969    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:49:03.490145 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:03.490084    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 20:49:03.492545 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:03.492507    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:49:03.495554 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:03.495526    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cbx9w"]
Nov 03 20:49:03.580856 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:03.580821    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7cea823-f76b-4e8c-a730-73da97dada6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-cbx9w\" (UID: \"c7cea823-f76b-4e8c-a730-73da97dada6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cbx9w"
Nov 03 20:49:03.581074 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:03.580864    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7cea823-f76b-4e8c-a730-73da97dada6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-cbx9w\" (UID: \"c7cea823-f76b-4e8c-a730-73da97dada6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cbx9w"
Nov 03 20:49:03.581074 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:03.581023    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwt6p\" (UniqueName: \"kubernetes.io/projected/c7cea823-f76b-4e8c-a730-73da97dada6f-kube-api-access-cwt6p\") pod \"maas-api-revoke-sub-452b6d7b-cbx9w\" (UID: \"c7cea823-f76b-4e8c-a730-73da97dada6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cbx9w"
Nov 03 20:49:03.682105 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:03.682050    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7cea823-f76b-4e8c-a730-73da97dada6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-cbx9w\" (UID: \"c7cea823-f76b-4e8c-a730-73da97dada6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cbx9w"
Nov 03 20:49:03.682380 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:03.682151    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwt6p\" (UniqueName: \"kubernetes.io/projected/c7cea823-f76b-4e8c-a730-73da97dada6f-kube-api-access-cwt6p\") pod \"maas-api-revoke-sub-452b6d7b-cbx9w\" (UID: \"c7cea823-f76b-4e8c-a730-73da97dada6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cbx9w"
Nov 03 20:49:03.682380 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:03.682260    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7cea823-f76b-4e8c-a730-73da97dada6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-cbx9w\" (UID: \"c7cea823-f76b-4e8c-a730-73da97dada6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cbx9w"
Nov 03 20:49:03.682845 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:03.682816    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7cea823-f76b-4e8c-a730-73da97dada6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-cbx9w\" (UID: \"c7cea823-f76b-4e8c-a730-73da97dada6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cbx9w"
Nov 03 20:49:03.684788 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:03.684768    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7cea823-f76b-4e8c-a730-73da97dada6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-cbx9w\" (UID: \"c7cea823-f76b-4e8c-a730-73da97dada6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cbx9w"
Nov 03 20:49:03.694533 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:03.694507    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwt6p\" (UniqueName: \"kubernetes.io/projected/c7cea823-f76b-4e8c-a730-73da97dada6f-kube-api-access-cwt6p\") pod \"maas-api-revoke-sub-452b6d7b-cbx9w\" (UID: \"c7cea823-f76b-4e8c-a730-73da97dada6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cbx9w"
Nov 03 20:49:03.801169 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:03.801131    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cbx9w"
Nov 03 20:49:04.080101 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:04.080070    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cbx9w"]
Nov 03 20:49:04.082760 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:49:04.082728    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7cea823_f76b_4e8c_a730_73da97dada6f.slice/crio-7661c288299bee649b41ad857d837ffec697dfbc0851781281b719d1c2b550e8 WatchSource:0}: Error finding container 7661c288299bee649b41ad857d837ffec697dfbc0851781281b719d1c2b550e8: Status 404 returned error can't find the container with id 7661c288299bee649b41ad857d837ffec697dfbc0851781281b719d1c2b550e8
Nov 03 20:49:04.221258 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:04.221218    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:49:05.067972 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:05.067933    5591 generic.go:356] "Generic (PLEG): container finished" podID="c7cea823-f76b-4e8c-a730-73da97dada6f" containerID="1c86e5bc8f8ce72d88f01b1f5660528b3a866b99e3628534dfffbe4e95642fe0" exitCode=0
Nov 03 20:49:05.068444 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:05.068110    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cbx9w" event={"ID":"c7cea823-f76b-4e8c-a730-73da97dada6f","Type":"ContainerDied","Data":"1c86e5bc8f8ce72d88f01b1f5660528b3a866b99e3628534dfffbe4e95642fe0"}
Nov 03 20:49:05.068444 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:05.068164    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cbx9w" event={"ID":"c7cea823-f76b-4e8c-a730-73da97dada6f","Type":"ContainerStarted","Data":"7661c288299bee649b41ad857d837ffec697dfbc0851781281b719d1c2b550e8"}
Nov 03 20:49:06.466545 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:06.466513    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cbx9w"
Nov 03 20:49:06.650703 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:06.650658    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7cea823-f76b-4e8c-a730-73da97dada6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7cea823-f76b-4e8c-a730-73da97dada6f-maas-api-xks-ca\") pod \"c7cea823-f76b-4e8c-a730-73da97dada6f\" (UID: \"c7cea823-f76b-4e8c-a730-73da97dada6f\") "
Nov 03 20:49:06.651082 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:06.651048    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7cea823-f76b-4e8c-a730-73da97dada6f-kube-api-access-cwt6p\" (UniqueName: \"kubernetes.io/projected/c7cea823-f76b-4e8c-a730-73da97dada6f-kube-api-access-cwt6p\") pod \"c7cea823-f76b-4e8c-a730-73da97dada6f\" (UID: \"c7cea823-f76b-4e8c-a730-73da97dada6f\") "
Nov 03 20:49:06.657233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:06.657158    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7cea823-f76b-4e8c-a730-73da97dada6f-maas-api-xks-ca" pod "c7cea823-f76b-4e8c-a730-73da97dada6f" (UID: "c7cea823-f76b-4e8c-a730-73da97dada6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:49:06.658861 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:06.658267    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7cea823-f76b-4e8c-a730-73da97dada6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7cea823-f76b-4e8c-a730-73da97dada6f-maas-api-openshift-ca\") pod \"c7cea823-f76b-4e8c-a730-73da97dada6f\" (UID: \"c7cea823-f76b-4e8c-a730-73da97dada6f\") "
Nov 03 20:49:06.659479 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:06.659431    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7cea823-f76b-4e8c-a730-73da97dada6f-maas-api-openshift-ca" pod "c7cea823-f76b-4e8c-a730-73da97dada6f" (UID: "c7cea823-f76b-4e8c-a730-73da97dada6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:49:06.664960 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:06.664902    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7cea823-f76b-4e8c-a730-73da97dada6f-kube-api-access-cwt6p" pod "c7cea823-f76b-4e8c-a730-73da97dada6f" (UID: "c7cea823-f76b-4e8c-a730-73da97dada6f"). InnerVolumeSpecName "kube-api-access-cwt6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:49:06.760313 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:06.760119    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7cea823-f76b-4e8c-a730-73da97dada6f-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:06.760313 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:06.760158    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwt6p\" (UniqueName: \"kubernetes.io/projected/c7cea823-f76b-4e8c-a730-73da97dada6f-kube-api-access-cwt6p\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:06.760313 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:06.760172    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7cea823-f76b-4e8c-a730-73da97dada6f-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:07.103231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:07.101582    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cbx9w" event={"ID":"c7cea823-f76b-4e8c-a730-73da97dada6f","Type":"ContainerDied","Data":"7661c288299bee649b41ad857d837ffec697dfbc0851781281b719d1c2b550e8"}
Nov 03 20:49:07.103231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:07.101655    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7661c288299bee649b41ad857d837ffec697dfbc0851781281b719d1c2b550e8"
Nov 03 20:49:07.103231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:07.101737    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cbx9w"
Nov 03 20:49:15.819696 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:15.819661    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:49:17.432169 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:17.432126    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nhsds"]
Nov 03 20:49:17.451234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:17.433473    5591 cpu_manager.go:414] "containerMap: removing container" podUID="1b577df4-3ef8-4c04-8f9e-e454254267d6" containerName="istio-proxy"
Nov 03 20:49:17.452013 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:17.451982    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b577df4-3ef8-4c04-8f9e-e454254267d6" containerName="istio-proxy"
Nov 03 20:49:17.452291 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:17.452271    5591 cpu_manager.go:414] "containerMap: removing container" podUID="c7cea823-f76b-4e8c-a730-73da97dada6f" containerName="revoke-keys"
Nov 03 20:49:17.452409 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:17.452395    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7cea823-f76b-4e8c-a730-73da97dada6f" containerName="revoke-keys"
Nov 03 20:49:17.452691 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:17.452675    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7cea823-f76b-4e8c-a730-73da97dada6f" containerName="revoke-keys"
Nov 03 20:49:17.461532 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:17.461493    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nhsds"]
Nov 03 20:49:17.461944 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:17.461868    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nhsds"
Nov 03 20:49:17.468158 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:17.468130    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:49:17.468345 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:17.468248    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 20:49:17.468345 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:17.468278    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:49:17.468345 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:17.468344    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:49:17.570194 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:17.570151    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6852b9b4-89f9-4d67-abf4-a8d8042f1269-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-nhsds\" (UID: \"6852b9b4-89f9-4d67-abf4-a8d8042f1269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nhsds"
Nov 03 20:49:17.570194 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:17.570222    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6852b9b4-89f9-4d67-abf4-a8d8042f1269-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-nhsds\" (UID: \"6852b9b4-89f9-4d67-abf4-a8d8042f1269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nhsds"
Nov 03 20:49:17.570514 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:17.570361    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qphk\" (UniqueName: \"kubernetes.io/projected/6852b9b4-89f9-4d67-abf4-a8d8042f1269-kube-api-access-2qphk\") pod \"maas-api-revoke-sub-3e31a506-nhsds\" (UID: \"6852b9b4-89f9-4d67-abf4-a8d8042f1269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nhsds"
Nov 03 20:49:17.671411 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:17.671360    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qphk\" (UniqueName: \"kubernetes.io/projected/6852b9b4-89f9-4d67-abf4-a8d8042f1269-kube-api-access-2qphk\") pod \"maas-api-revoke-sub-3e31a506-nhsds\" (UID: \"6852b9b4-89f9-4d67-abf4-a8d8042f1269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nhsds"
Nov 03 20:49:17.671653 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:17.671460    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6852b9b4-89f9-4d67-abf4-a8d8042f1269-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-nhsds\" (UID: \"6852b9b4-89f9-4d67-abf4-a8d8042f1269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nhsds"
Nov 03 20:49:17.671653 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:17.671484    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6852b9b4-89f9-4d67-abf4-a8d8042f1269-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-nhsds\" (UID: \"6852b9b4-89f9-4d67-abf4-a8d8042f1269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nhsds"
Nov 03 20:49:17.672177 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:17.672150    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6852b9b4-89f9-4d67-abf4-a8d8042f1269-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-nhsds\" (UID: \"6852b9b4-89f9-4d67-abf4-a8d8042f1269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nhsds"
Nov 03 20:49:17.674136 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:17.674115    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6852b9b4-89f9-4d67-abf4-a8d8042f1269-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-nhsds\" (UID: \"6852b9b4-89f9-4d67-abf4-a8d8042f1269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nhsds"
Nov 03 20:49:17.700138 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:17.700053    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qphk\" (UniqueName: \"kubernetes.io/projected/6852b9b4-89f9-4d67-abf4-a8d8042f1269-kube-api-access-2qphk\") pod \"maas-api-revoke-sub-3e31a506-nhsds\" (UID: \"6852b9b4-89f9-4d67-abf4-a8d8042f1269\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nhsds"
Nov 03 20:49:17.810738 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:17.810399    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nhsds"
Nov 03 20:49:18.236333 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:18.221341    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nhsds"]
Nov 03 20:49:19.168868 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:19.168829    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nhsds" event={"ID":"6852b9b4-89f9-4d67-abf4-a8d8042f1269","Type":"ContainerStarted","Data":"37accab11ae456f62f0689fa7b91154d3ea1d768715d5bcce282dfdcb8482748"}
Nov 03 20:49:19.169328 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:19.168922    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nhsds" event={"ID":"6852b9b4-89f9-4d67-abf4-a8d8042f1269","Type":"ContainerStarted","Data":"4a8e7e638f7aa6e8c70f5657957cfb427c53a16a8346b462ff76f6b7dc4b214b"}
Nov 03 20:49:19.191751 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:19.190367    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nhsds" podStartSLOduration=2.190347036 podStartE2EDuration="2.190347036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:49:19.188589752 +0000 UTC m=+4307.960882773" watchObservedRunningTime="2026-11-03 20:49:19.190347036 +0000 UTC m=+4307.962640054"
Nov 03 20:49:20.183932 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:20.183884    5591 generic.go:356] "Generic (PLEG): container finished" podID="6852b9b4-89f9-4d67-abf4-a8d8042f1269" containerID="37accab11ae456f62f0689fa7b91154d3ea1d768715d5bcce282dfdcb8482748" exitCode=0
Nov 03 20:49:20.184557 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:20.183965    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nhsds" event={"ID":"6852b9b4-89f9-4d67-abf4-a8d8042f1269","Type":"ContainerDied","Data":"37accab11ae456f62f0689fa7b91154d3ea1d768715d5bcce282dfdcb8482748"}
Nov 03 20:49:21.308755 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:21.308717    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:49:21.565477 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:21.564688    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nhsds"
Nov 03 20:49:21.666688 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:21.666644    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6852b9b4-89f9-4d67-abf4-a8d8042f1269-kube-api-access-2qphk\" (UniqueName: \"kubernetes.io/projected/6852b9b4-89f9-4d67-abf4-a8d8042f1269-kube-api-access-2qphk\") pod \"6852b9b4-89f9-4d67-abf4-a8d8042f1269\" (UID: \"6852b9b4-89f9-4d67-abf4-a8d8042f1269\") "
Nov 03 20:49:21.666916 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:21.666752    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6852b9b4-89f9-4d67-abf4-a8d8042f1269-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6852b9b4-89f9-4d67-abf4-a8d8042f1269-maas-api-openshift-ca\") pod \"6852b9b4-89f9-4d67-abf4-a8d8042f1269\" (UID: \"6852b9b4-89f9-4d67-abf4-a8d8042f1269\") "
Nov 03 20:49:21.666916 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:21.666796    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6852b9b4-89f9-4d67-abf4-a8d8042f1269-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6852b9b4-89f9-4d67-abf4-a8d8042f1269-maas-api-xks-ca\") pod \"6852b9b4-89f9-4d67-abf4-a8d8042f1269\" (UID: \"6852b9b4-89f9-4d67-abf4-a8d8042f1269\") "
Nov 03 20:49:21.667978 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:21.667948    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6852b9b4-89f9-4d67-abf4-a8d8042f1269-maas-api-openshift-ca" pod "6852b9b4-89f9-4d67-abf4-a8d8042f1269" (UID: "6852b9b4-89f9-4d67-abf4-a8d8042f1269"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:49:21.670067 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:21.670033    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6852b9b4-89f9-4d67-abf4-a8d8042f1269-maas-api-xks-ca" pod "6852b9b4-89f9-4d67-abf4-a8d8042f1269" (UID: "6852b9b4-89f9-4d67-abf4-a8d8042f1269"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:49:21.670389 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:21.670365    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6852b9b4-89f9-4d67-abf4-a8d8042f1269-kube-api-access-2qphk" pod "6852b9b4-89f9-4d67-abf4-a8d8042f1269" (UID: "6852b9b4-89f9-4d67-abf4-a8d8042f1269"). InnerVolumeSpecName "kube-api-access-2qphk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:49:21.768385 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:21.768345    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qphk\" (UniqueName: \"kubernetes.io/projected/6852b9b4-89f9-4d67-abf4-a8d8042f1269-kube-api-access-2qphk\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:21.768385 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:21.768383    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6852b9b4-89f9-4d67-abf4-a8d8042f1269-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:21.768605 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:21.768394    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6852b9b4-89f9-4d67-abf4-a8d8042f1269-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:49:22.211976 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:22.211748    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nhsds" event={"ID":"6852b9b4-89f9-4d67-abf4-a8d8042f1269","Type":"ContainerDied","Data":"4a8e7e638f7aa6e8c70f5657957cfb427c53a16a8346b462ff76f6b7dc4b214b"}
Nov 03 20:49:22.211976 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:22.211817    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a8e7e638f7aa6e8c70f5657957cfb427c53a16a8346b462ff76f6b7dc4b214b"
Nov 03 20:49:22.216339 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:22.212434    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nhsds"
Nov 03 20:49:28.296850 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:28.296805    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:49:28.735612 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:49:28.735515    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:49:32.555236 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:32.555177    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:49:45.936965 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:45.936878    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:49:48.492528 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:48.492492    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-c44c86556-c65mk"]
Nov 03 20:49:48.493099 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:48.492958    5591 cpu_manager.go:414] "containerMap: removing container" podUID="6852b9b4-89f9-4d67-abf4-a8d8042f1269" containerName="revoke-keys"
Nov 03 20:49:48.493099 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:48.492970    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6852b9b4-89f9-4d67-abf4-a8d8042f1269" containerName="revoke-keys"
Nov 03 20:49:48.493099 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:48.493065    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="6852b9b4-89f9-4d67-abf4-a8d8042f1269" containerName="revoke-keys"
Nov 03 20:49:48.496395 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:48.496380    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-c44c86556-c65mk"
Nov 03 20:49:48.504716 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:48.504689    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-c44c86556-c65mk"]
Nov 03 20:49:48.573456 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:48.573418    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kppl5\" (UniqueName: \"kubernetes.io/projected/4a78f787-3921-4287-a438-40ff5ebc7703-kube-api-access-kppl5\") pod \"payload-pre-processing-c44c86556-c65mk\" (UID: \"4a78f787-3921-4287-a438-40ff5ebc7703\") " pod="openshift-ingress/payload-pre-processing-c44c86556-c65mk"
Nov 03 20:49:48.573456 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:48.573460    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a78f787-3921-4287-a438-40ff5ebc7703-plugins-config-volume\") pod \"payload-pre-processing-c44c86556-c65mk\" (UID: \"4a78f787-3921-4287-a438-40ff5ebc7703\") " pod="openshift-ingress/payload-pre-processing-c44c86556-c65mk"
Nov 03 20:49:48.673909 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:48.673867    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kppl5\" (UniqueName: \"kubernetes.io/projected/4a78f787-3921-4287-a438-40ff5ebc7703-kube-api-access-kppl5\") pod \"payload-pre-processing-c44c86556-c65mk\" (UID: \"4a78f787-3921-4287-a438-40ff5ebc7703\") " pod="openshift-ingress/payload-pre-processing-c44c86556-c65mk"
Nov 03 20:49:48.673909 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:48.673913    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a78f787-3921-4287-a438-40ff5ebc7703-plugins-config-volume\") pod \"payload-pre-processing-c44c86556-c65mk\" (UID: \"4a78f787-3921-4287-a438-40ff5ebc7703\") " pod="openshift-ingress/payload-pre-processing-c44c86556-c65mk"
Nov 03 20:49:48.674744 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:48.674721    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a78f787-3921-4287-a438-40ff5ebc7703-plugins-config-volume\") pod \"payload-pre-processing-c44c86556-c65mk\" (UID: \"4a78f787-3921-4287-a438-40ff5ebc7703\") " pod="openshift-ingress/payload-pre-processing-c44c86556-c65mk"
Nov 03 20:49:48.687241 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:48.687195    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kppl5\" (UniqueName: \"kubernetes.io/projected/4a78f787-3921-4287-a438-40ff5ebc7703-kube-api-access-kppl5\") pod \"payload-pre-processing-c44c86556-c65mk\" (UID: \"4a78f787-3921-4287-a438-40ff5ebc7703\") " pod="openshift-ingress/payload-pre-processing-c44c86556-c65mk"
Nov 03 20:49:48.811037 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:48.810993    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-c44c86556-c65mk"
Nov 03 20:49:49.087713 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:49.087683    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-c44c86556-c65mk"]
Nov 03 20:49:49.090434 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:49:49.090401    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a78f787_3921_4287_a438_40ff5ebc7703.slice/crio-2587bce07718fc271978759c927c4e436f806495cb064193538477c9a58a3445 WatchSource:0}: Error finding container 2587bce07718fc271978759c927c4e436f806495cb064193538477c9a58a3445: Status 404 returned error can't find the container with id 2587bce07718fc271978759c927c4e436f806495cb064193538477c9a58a3445
Nov 03 20:49:49.359106 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:49.359011    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-c44c86556-c65mk" event={"ID":"4a78f787-3921-4287-a438-40ff5ebc7703","Type":"ContainerStarted","Data":"b98c36f07cc872ffd5f41aeb04cb2aa5d7cc34d216bf3070d0f4aacd51c7291e"}
Nov 03 20:49:49.359106 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:49.359069    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-c44c86556-c65mk" event={"ID":"4a78f787-3921-4287-a438-40ff5ebc7703","Type":"ContainerStarted","Data":"2587bce07718fc271978759c927c4e436f806495cb064193538477c9a58a3445"}
Nov 03 20:49:49.359106 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:49.359090    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-c44c86556-c65mk"
Nov 03 20:49:49.378044 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:49:49.377992    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-c44c86556-c65mk" podStartSLOduration=1.377978486 podStartE2EDuration="1.377978486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:49:49.376671807 +0000 UTC m=+4338.148964826" watchObservedRunningTime="2026-11-03 20:49:49.377978486 +0000 UTC m=+4338.150271503"
Nov 03 20:49:58.723614 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:49:58.723567    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:50:00.367431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:00.367397    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-c44c86556-c65mk"
Nov 03 20:50:00.416220 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:00.416166    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-45rfv"]
Nov 03 20:50:02.178857 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:02.178820    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-9c77b64f4-hdd5s"]
Nov 03 20:50:02.182719 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:02.182694    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-9c77b64f4-hdd5s"
Nov 03 20:50:02.191917 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:02.191890    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-9c77b64f4-hdd5s"]
Nov 03 20:50:02.211083 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:02.211041    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a2db403-e4b9-4e2b-ae5b-cf620bb89196-plugins-config-volume\") pod \"payload-processing-9c77b64f4-hdd5s\" (UID: \"3a2db403-e4b9-4e2b-ae5b-cf620bb89196\") " pod="openshift-ingress/payload-processing-9c77b64f4-hdd5s"
Nov 03 20:50:02.211330 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:02.211224    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8wqj\" (UniqueName: \"kubernetes.io/projected/3a2db403-e4b9-4e2b-ae5b-cf620bb89196-kube-api-access-k8wqj\") pod \"payload-processing-9c77b64f4-hdd5s\" (UID: \"3a2db403-e4b9-4e2b-ae5b-cf620bb89196\") " pod="openshift-ingress/payload-processing-9c77b64f4-hdd5s"
Nov 03 20:50:02.312683 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:02.312639    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8wqj\" (UniqueName: \"kubernetes.io/projected/3a2db403-e4b9-4e2b-ae5b-cf620bb89196-kube-api-access-k8wqj\") pod \"payload-processing-9c77b64f4-hdd5s\" (UID: \"3a2db403-e4b9-4e2b-ae5b-cf620bb89196\") " pod="openshift-ingress/payload-processing-9c77b64f4-hdd5s"
Nov 03 20:50:02.312896 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:02.312750    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a2db403-e4b9-4e2b-ae5b-cf620bb89196-plugins-config-volume\") pod \"payload-processing-9c77b64f4-hdd5s\" (UID: \"3a2db403-e4b9-4e2b-ae5b-cf620bb89196\") " pod="openshift-ingress/payload-processing-9c77b64f4-hdd5s"
Nov 03 20:50:02.313512 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:02.313490    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a2db403-e4b9-4e2b-ae5b-cf620bb89196-plugins-config-volume\") pod \"payload-processing-9c77b64f4-hdd5s\" (UID: \"3a2db403-e4b9-4e2b-ae5b-cf620bb89196\") " pod="openshift-ingress/payload-processing-9c77b64f4-hdd5s"
Nov 03 20:50:02.325733 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:02.325701    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8wqj\" (UniqueName: \"kubernetes.io/projected/3a2db403-e4b9-4e2b-ae5b-cf620bb89196-kube-api-access-k8wqj\") pod \"payload-processing-9c77b64f4-hdd5s\" (UID: \"3a2db403-e4b9-4e2b-ae5b-cf620bb89196\") " pod="openshift-ingress/payload-processing-9c77b64f4-hdd5s"
Nov 03 20:50:02.497980 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:02.497883    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-9c77b64f4-hdd5s"
Nov 03 20:50:02.776357 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:02.776115    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-9c77b64f4-hdd5s"]
Nov 03 20:50:02.778920 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:50:02.778891    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a2db403_e4b9_4e2b_ae5b_cf620bb89196.slice/crio-360fe592bd6889dc3ba16f8e63fb32f973520b02cc9da5cb1fa28d85cc3da6f3 WatchSource:0}: Error finding container 360fe592bd6889dc3ba16f8e63fb32f973520b02cc9da5cb1fa28d85cc3da6f3: Status 404 returned error can't find the container with id 360fe592bd6889dc3ba16f8e63fb32f973520b02cc9da5cb1fa28d85cc3da6f3
Nov 03 20:50:03.432478 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:03.432443    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-9c77b64f4-hdd5s" event={"ID":"3a2db403-e4b9-4e2b-ae5b-cf620bb89196","Type":"ContainerStarted","Data":"caf142ac0e2fca0a9d21681edfbbdd415fa78119264529c53c4c01b71d3c212d"}
Nov 03 20:50:03.432969 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:03.432500    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-9c77b64f4-hdd5s" event={"ID":"3a2db403-e4b9-4e2b-ae5b-cf620bb89196","Type":"ContainerStarted","Data":"360fe592bd6889dc3ba16f8e63fb32f973520b02cc9da5cb1fa28d85cc3da6f3"}
Nov 03 20:50:03.432969 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:03.432526    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-9c77b64f4-hdd5s"
Nov 03 20:50:03.452190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:03.452135    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-9c77b64f4-hdd5s" podStartSLOduration=1.452119097 podStartE2EDuration="1.452119097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:03.450533189 +0000 UTC m=+4352.222826204" watchObservedRunningTime="2026-11-03 20:50:03.452119097 +0000 UTC m=+4352.224412108"
Nov 03 20:50:05.417191 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:05.417148    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-45rfv" podUID="fc67cda2-1534-4223-be48-0130220d6360" containerName="payload-pre-processing" containerID="cri-o://74e343698c5915fb2a6daf91df1dbe22c16e3b6fc5f2f4918b78c0d48ff099e4" gracePeriod=25
Nov 03 20:50:05.668236 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:05.668162    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-45rfv"
Nov 03 20:50:05.747424 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:05.747388    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc67cda2-1534-4223-be48-0130220d6360-kube-api-access-jr967\" (UniqueName: \"kubernetes.io/projected/fc67cda2-1534-4223-be48-0130220d6360-kube-api-access-jr967\") pod \"fc67cda2-1534-4223-be48-0130220d6360\" (UID: \"fc67cda2-1534-4223-be48-0130220d6360\") "
Nov 03 20:50:05.747623 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:05.747467    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc67cda2-1534-4223-be48-0130220d6360-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc67cda2-1534-4223-be48-0130220d6360-plugins-config-volume\") pod \"fc67cda2-1534-4223-be48-0130220d6360\" (UID: \"fc67cda2-1534-4223-be48-0130220d6360\") "
Nov 03 20:50:05.747787 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:05.747764    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc67cda2-1534-4223-be48-0130220d6360-plugins-config-volume" pod "fc67cda2-1534-4223-be48-0130220d6360" (UID: "fc67cda2-1534-4223-be48-0130220d6360"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:50:05.749691 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:05.749668    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc67cda2-1534-4223-be48-0130220d6360-kube-api-access-jr967" pod "fc67cda2-1534-4223-be48-0130220d6360" (UID: "fc67cda2-1534-4223-be48-0130220d6360"). InnerVolumeSpecName "kube-api-access-jr967". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:50:05.849234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:05.849162    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr967\" (UniqueName: \"kubernetes.io/projected/fc67cda2-1534-4223-be48-0130220d6360-kube-api-access-jr967\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:05.849234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:05.849197    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc67cda2-1534-4223-be48-0130220d6360-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:06.447948 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:06.447908    5591 generic.go:356] "Generic (PLEG): container finished" podID="fc67cda2-1534-4223-be48-0130220d6360" containerID="74e343698c5915fb2a6daf91df1dbe22c16e3b6fc5f2f4918b78c0d48ff099e4" exitCode=0
Nov 03 20:50:06.448497 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:06.447958    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-45rfv"
Nov 03 20:50:06.448497 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:06.447995    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-45rfv" event={"ID":"fc67cda2-1534-4223-be48-0130220d6360","Type":"ContainerDied","Data":"74e343698c5915fb2a6daf91df1dbe22c16e3b6fc5f2f4918b78c0d48ff099e4"}
Nov 03 20:50:06.448497 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:06.448051    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-45rfv" event={"ID":"fc67cda2-1534-4223-be48-0130220d6360","Type":"ContainerDied","Data":"9e205e75018541a355d7387af293a560e32e39053879db453a48b4abc862837c"}
Nov 03 20:50:06.448497 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:06.448072    5591 scope.go:122] "RemoveContainer" containerID="74e343698c5915fb2a6daf91df1dbe22c16e3b6fc5f2f4918b78c0d48ff099e4"
Nov 03 20:50:06.458125 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:06.458109    5591 scope.go:122] "RemoveContainer" containerID="74e343698c5915fb2a6daf91df1dbe22c16e3b6fc5f2f4918b78c0d48ff099e4"
Nov 03 20:50:06.458414 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:50:06.458395    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74e343698c5915fb2a6daf91df1dbe22c16e3b6fc5f2f4918b78c0d48ff099e4\": container with ID starting with 74e343698c5915fb2a6daf91df1dbe22c16e3b6fc5f2f4918b78c0d48ff099e4 not found: ID does not exist" containerID="74e343698c5915fb2a6daf91df1dbe22c16e3b6fc5f2f4918b78c0d48ff099e4"
Nov 03 20:50:06.458498 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:06.458425    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e343698c5915fb2a6daf91df1dbe22c16e3b6fc5f2f4918b78c0d48ff099e4"} err="failed to get container status \"74e343698c5915fb2a6daf91df1dbe22c16e3b6fc5f2f4918b78c0d48ff099e4\": rpc error: code = NotFound desc = could not find container \"74e343698c5915fb2a6daf91df1dbe22c16e3b6fc5f2f4918b78c0d48ff099e4\": container with ID starting with 74e343698c5915fb2a6daf91df1dbe22c16e3b6fc5f2f4918b78c0d48ff099e4 not found: ID does not exist"
Nov 03 20:50:06.470710 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:06.470686    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-45rfv"]
Nov 03 20:50:06.475020 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:06.474997    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-45rfv"]
Nov 03 20:50:07.347922 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:07.347881    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc67cda2-1534-4223-be48-0130220d6360" path="/var/lib/kubelet/pods/fc67cda2-1534-4223-be48-0130220d6360/volumes"
Nov 03 20:50:14.440866 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:14.440821    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-9c77b64f4-hdd5s"
Nov 03 20:50:14.499949 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:14.499913    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-x84lg"]
Nov 03 20:50:16.436392 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:16.436353    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-68d58ff5c7-z4hlv"]
Nov 03 20:50:16.436825 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:16.436805    5591 cpu_manager.go:414] "containerMap: removing container" podUID="fc67cda2-1534-4223-be48-0130220d6360" containerName="payload-pre-processing"
Nov 03 20:50:16.436825 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:16.436816    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc67cda2-1534-4223-be48-0130220d6360" containerName="payload-pre-processing"
Nov 03 20:50:16.436918 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:16.436895    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc67cda2-1534-4223-be48-0130220d6360" containerName="payload-pre-processing"
Nov 03 20:50:16.440118 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:16.440088    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-68d58ff5c7-z4hlv"
Nov 03 20:50:16.450252 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:16.450197    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-68d58ff5c7-z4hlv"]
Nov 03 20:50:16.561240 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:16.561187    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990d3b87-4fac-45c8-862b-f8dca1606467-plugins-config-volume\") pod \"payload-processing-68d58ff5c7-z4hlv\" (UID: \"990d3b87-4fac-45c8-862b-f8dca1606467\") " pod="openshift-ingress/payload-processing-68d58ff5c7-z4hlv"
Nov 03 20:50:16.561446 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:16.561338    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmnb2\" (UniqueName: \"kubernetes.io/projected/990d3b87-4fac-45c8-862b-f8dca1606467-kube-api-access-pmnb2\") pod \"payload-processing-68d58ff5c7-z4hlv\" (UID: \"990d3b87-4fac-45c8-862b-f8dca1606467\") " pod="openshift-ingress/payload-processing-68d58ff5c7-z4hlv"
Nov 03 20:50:16.662118 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:16.662082    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990d3b87-4fac-45c8-862b-f8dca1606467-plugins-config-volume\") pod \"payload-processing-68d58ff5c7-z4hlv\" (UID: \"990d3b87-4fac-45c8-862b-f8dca1606467\") " pod="openshift-ingress/payload-processing-68d58ff5c7-z4hlv"
Nov 03 20:50:16.662337 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:16.662187    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmnb2\" (UniqueName: \"kubernetes.io/projected/990d3b87-4fac-45c8-862b-f8dca1606467-kube-api-access-pmnb2\") pod \"payload-processing-68d58ff5c7-z4hlv\" (UID: \"990d3b87-4fac-45c8-862b-f8dca1606467\") " pod="openshift-ingress/payload-processing-68d58ff5c7-z4hlv"
Nov 03 20:50:16.662910 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:16.662891    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990d3b87-4fac-45c8-862b-f8dca1606467-plugins-config-volume\") pod \"payload-processing-68d58ff5c7-z4hlv\" (UID: \"990d3b87-4fac-45c8-862b-f8dca1606467\") " pod="openshift-ingress/payload-processing-68d58ff5c7-z4hlv"
Nov 03 20:50:16.674767 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:16.674740    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmnb2\" (UniqueName: \"kubernetes.io/projected/990d3b87-4fac-45c8-862b-f8dca1606467-kube-api-access-pmnb2\") pod \"payload-processing-68d58ff5c7-z4hlv\" (UID: \"990d3b87-4fac-45c8-862b-f8dca1606467\") " pod="openshift-ingress/payload-processing-68d58ff5c7-z4hlv"
Nov 03 20:50:16.755081 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:16.754990    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-68d58ff5c7-z4hlv"
Nov 03 20:50:17.033517 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:17.033489    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-68d58ff5c7-z4hlv"]
Nov 03 20:50:17.036221 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:50:17.036179    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990d3b87_4fac_45c8_862b_f8dca1606467.slice/crio-a6c705fe59ed5d1866637afb167d629a1e5cf58b4d139c01b240349d0998b6ba WatchSource:0}: Error finding container a6c705fe59ed5d1866637afb167d629a1e5cf58b4d139c01b240349d0998b6ba: Status 404 returned error can't find the container with id a6c705fe59ed5d1866637afb167d629a1e5cf58b4d139c01b240349d0998b6ba
Nov 03 20:50:17.507778 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:17.507680    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-68d58ff5c7-z4hlv" event={"ID":"990d3b87-4fac-45c8-862b-f8dca1606467","Type":"ContainerStarted","Data":"817295f1c0524783c010ad5f4d5be26b6ef182faac2dd89cae10e887bf1e5ce2"}
Nov 03 20:50:17.508139 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:17.507778    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-68d58ff5c7-z4hlv" event={"ID":"990d3b87-4fac-45c8-862b-f8dca1606467","Type":"ContainerStarted","Data":"a6c705fe59ed5d1866637afb167d629a1e5cf58b4d139c01b240349d0998b6ba"}
Nov 03 20:50:17.508139 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:17.507826    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-68d58ff5c7-z4hlv"
Nov 03 20:50:17.527169 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:17.527121    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-68d58ff5c7-z4hlv" podStartSLOduration=1.527103518 podStartE2EDuration="1.527103518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:17.526634083 +0000 UTC m=+4366.298927102" watchObservedRunningTime="2026-11-03 20:50:17.527103518 +0000 UTC m=+4366.299396537"
Nov 03 20:50:19.500647 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:19.500603    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-7bb9489676-x84lg" podUID="d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d" containerName="payload-processing" containerID="cri-o://9311d981c99f20f4a4988aebc309b5ab0ffb2ca28a0c511af908f2ac3d7cba0f" gracePeriod=25
Nov 03 20:50:19.754557 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:19.754498    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-x84lg"
Nov 03 20:50:19.896561 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:19.896527    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d-kube-api-access-8jsgj\" (UniqueName: \"kubernetes.io/projected/d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d-kube-api-access-8jsgj\") pod \"d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d\" (UID: \"d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d\") "
Nov 03 20:50:19.896759 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:19.896589    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d-plugins-config-volume\") pod \"d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d\" (UID: \"d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d\") "
Nov 03 20:50:19.897024 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:19.896995    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d-plugins-config-volume" pod "d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d" (UID: "d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:50:19.898926 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:19.898900    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d-kube-api-access-8jsgj" pod "d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d" (UID: "d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d"). InnerVolumeSpecName "kube-api-access-8jsgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:50:19.997850 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:19.997808    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jsgj\" (UniqueName: \"kubernetes.io/projected/d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d-kube-api-access-8jsgj\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:19.997850 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:19.997841    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:20.524808 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:20.524774    5591 generic.go:356] "Generic (PLEG): container finished" podID="d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d" containerID="9311d981c99f20f4a4988aebc309b5ab0ffb2ca28a0c511af908f2ac3d7cba0f" exitCode=0
Nov 03 20:50:20.525283 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:20.524827    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-x84lg"
Nov 03 20:50:20.525283 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:20.524872    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-x84lg" event={"ID":"d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d","Type":"ContainerDied","Data":"9311d981c99f20f4a4988aebc309b5ab0ffb2ca28a0c511af908f2ac3d7cba0f"}
Nov 03 20:50:20.525283 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:20.524925    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-x84lg" event={"ID":"d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d","Type":"ContainerDied","Data":"adf51a1c316450a9a00e62c7026d053ae5588b54a13ab43fcd72146a4c5ca6d7"}
Nov 03 20:50:20.525283 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:20.524946    5591 scope.go:122] "RemoveContainer" containerID="9311d981c99f20f4a4988aebc309b5ab0ffb2ca28a0c511af908f2ac3d7cba0f"
Nov 03 20:50:20.534939 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:20.534920    5591 scope.go:122] "RemoveContainer" containerID="9311d981c99f20f4a4988aebc309b5ab0ffb2ca28a0c511af908f2ac3d7cba0f"
Nov 03 20:50:20.535188 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:50:20.535165    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9311d981c99f20f4a4988aebc309b5ab0ffb2ca28a0c511af908f2ac3d7cba0f\": container with ID starting with 9311d981c99f20f4a4988aebc309b5ab0ffb2ca28a0c511af908f2ac3d7cba0f not found: ID does not exist" containerID="9311d981c99f20f4a4988aebc309b5ab0ffb2ca28a0c511af908f2ac3d7cba0f"
Nov 03 20:50:20.535290 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:20.535199    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9311d981c99f20f4a4988aebc309b5ab0ffb2ca28a0c511af908f2ac3d7cba0f"} err="failed to get container status \"9311d981c99f20f4a4988aebc309b5ab0ffb2ca28a0c511af908f2ac3d7cba0f\": rpc error: code = NotFound desc = could not find container \"9311d981c99f20f4a4988aebc309b5ab0ffb2ca28a0c511af908f2ac3d7cba0f\": container with ID starting with 9311d981c99f20f4a4988aebc309b5ab0ffb2ca28a0c511af908f2ac3d7cba0f not found: ID does not exist"
Nov 03 20:50:20.546108 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:20.546083    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-x84lg"]
Nov 03 20:50:20.550190 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:20.550163    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-x84lg"]
Nov 03 20:50:21.348075 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:21.348042    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d" path="/var/lib/kubelet/pods/d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d/volumes"
Nov 03 20:50:28.451260 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:28.451225    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-c44c86556-c65mk"]
Nov 03 20:50:28.516263 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:28.516234    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-68d58ff5c7-z4hlv"
Nov 03 20:50:28.562124 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:28.562092    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-9c77b64f4-hdd5s"]
Nov 03 20:50:28.724033 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:50:28.723939    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:50:31.223193 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:31.223157    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9"]
Nov 03 20:50:31.223682 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:31.223665    5591 cpu_manager.go:414] "containerMap: removing container" podUID="d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d" containerName="payload-processing"
Nov 03 20:50:31.223767 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:31.223685    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d" containerName="payload-processing"
Nov 03 20:50:31.223843 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:31.223821    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2672d13-b0a8-4e7e-81f9-4b0a04e69e1d" containerName="payload-processing"
Nov 03 20:50:31.228591 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:31.228571    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9"
Nov 03 20:50:31.237684 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:31.237659    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 20:50:31.237883 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:31.237860    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 20:50:31.237883 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:31.237814    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 20:50:31.238067 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:31.237942    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 20:50:31.238067 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:31.237949    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\""
Nov 03 20:50:31.238067 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:31.237683    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-4c86r\""
Nov 03 20:50:31.238426 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:31.238405    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9"]
Nov 03 20:50:31.298954 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:31.298923    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c08bc5b-267a-4356-93c3-c2629a24a486-plugins-config-volume\") pod \"payload-processing-external-model-7fd6967f4c-t5nx9\" (UID: \"6c08bc5b-267a-4356-93c3-c2629a24a486\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9"
Nov 03 20:50:31.298954 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:31.298962    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c08bc5b-267a-4356-93c3-c2629a24a486-routing-overlay-volume\") pod \"payload-processing-external-model-7fd6967f4c-t5nx9\" (UID: \"6c08bc5b-267a-4356-93c3-c2629a24a486\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9"
Nov 03 20:50:31.299237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:31.298995    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c08bc5b-267a-4356-93c3-c2629a24a486-provider-credentials\") pod \"payload-processing-external-model-7fd6967f4c-t5nx9\" (UID: \"6c08bc5b-267a-4356-93c3-c2629a24a486\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9"
Nov 03 20:50:31.299237 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:31.299138    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2rs9\" (UniqueName: \"kubernetes.io/projected/6c08bc5b-267a-4356-93c3-c2629a24a486-kube-api-access-p2rs9\") pod \"payload-processing-external-model-7fd6967f4c-t5nx9\" (UID: \"6c08bc5b-267a-4356-93c3-c2629a24a486\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9"
Nov 03 20:50:31.399892 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:31.399850    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2rs9\" (UniqueName: \"kubernetes.io/projected/6c08bc5b-267a-4356-93c3-c2629a24a486-kube-api-access-p2rs9\") pod \"payload-processing-external-model-7fd6967f4c-t5nx9\" (UID: \"6c08bc5b-267a-4356-93c3-c2629a24a486\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9"
Nov 03 20:50:31.400116 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:31.399925    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c08bc5b-267a-4356-93c3-c2629a24a486-plugins-config-volume\") pod \"payload-processing-external-model-7fd6967f4c-t5nx9\" (UID: \"6c08bc5b-267a-4356-93c3-c2629a24a486\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9"
Nov 03 20:50:31.400116 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:31.399975    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c08bc5b-267a-4356-93c3-c2629a24a486-routing-overlay-volume\") pod \"payload-processing-external-model-7fd6967f4c-t5nx9\" (UID: \"6c08bc5b-267a-4356-93c3-c2629a24a486\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9"
Nov 03 20:50:31.400116 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:31.400024    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c08bc5b-267a-4356-93c3-c2629a24a486-provider-credentials\") pod \"payload-processing-external-model-7fd6967f4c-t5nx9\" (UID: \"6c08bc5b-267a-4356-93c3-c2629a24a486\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9"
Nov 03 20:50:31.400723 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:31.400696    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c08bc5b-267a-4356-93c3-c2629a24a486-routing-overlay-volume\") pod \"payload-processing-external-model-7fd6967f4c-t5nx9\" (UID: \"6c08bc5b-267a-4356-93c3-c2629a24a486\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9"
Nov 03 20:50:31.401067 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:31.401038    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c08bc5b-267a-4356-93c3-c2629a24a486-plugins-config-volume\") pod \"payload-processing-external-model-7fd6967f4c-t5nx9\" (UID: \"6c08bc5b-267a-4356-93c3-c2629a24a486\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9"
Nov 03 20:50:31.402857 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:31.402833    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c08bc5b-267a-4356-93c3-c2629a24a486-provider-credentials\") pod \"payload-processing-external-model-7fd6967f4c-t5nx9\" (UID: \"6c08bc5b-267a-4356-93c3-c2629a24a486\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9"
Nov 03 20:50:31.412417 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:31.412381    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2rs9\" (UniqueName: \"kubernetes.io/projected/6c08bc5b-267a-4356-93c3-c2629a24a486-kube-api-access-p2rs9\") pod \"payload-processing-external-model-7fd6967f4c-t5nx9\" (UID: \"6c08bc5b-267a-4356-93c3-c2629a24a486\") " pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9"
Nov 03 20:50:31.544659 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:31.544628    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9"
Nov 03 20:50:31.821068 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:31.821040    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9"]
Nov 03 20:50:31.823335 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:50:31.823307    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c08bc5b_267a_4356_93c3_c2629a24a486.slice/crio-8b97e48441b6d1fe0bd65bb911947fb472cc00a87ed559768203f29479790e66 WatchSource:0}: Error finding container 8b97e48441b6d1fe0bd65bb911947fb472cc00a87ed559768203f29479790e66: Status 404 returned error can't find the container with id 8b97e48441b6d1fe0bd65bb911947fb472cc00a87ed559768203f29479790e66
Nov 03 20:50:32.442255 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:32.442221    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:50:32.592003 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:32.591960    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9" event={"ID":"6c08bc5b-267a-4356-93c3-c2629a24a486","Type":"ContainerStarted","Data":"b842a996fa7986b25dcc2c67f5d9b968868b52a68f9f23f93bb9b5f181543698"}
Nov 03 20:50:32.592233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:32.592014    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9" event={"ID":"6c08bc5b-267a-4356-93c3-c2629a24a486","Type":"ContainerStarted","Data":"8b97e48441b6d1fe0bd65bb911947fb472cc00a87ed559768203f29479790e66"}
Nov 03 20:50:32.592233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:32.592048    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9"
Nov 03 20:50:32.615119 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:32.615066    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9" podStartSLOduration=1.615053538 podStartE2EDuration="1.615053538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:32.612332677 +0000 UTC m=+4381.384625695" watchObservedRunningTime="2026-11-03 20:50:32.615053538 +0000 UTC m=+4381.387346557"
Nov 03 20:50:33.452029 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:33.451982    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-c44c86556-c65mk" podUID="4a78f787-3921-4287-a438-40ff5ebc7703" containerName="payload-pre-processing" containerID="cri-o://b98c36f07cc872ffd5f41aeb04cb2aa5d7cc34d216bf3070d0f4aacd51c7291e" gracePeriod=25
Nov 03 20:50:33.562936 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:33.562877    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-9c77b64f4-hdd5s" podUID="3a2db403-e4b9-4e2b-ae5b-cf620bb89196" containerName="payload-processing" containerID="cri-o://caf142ac0e2fca0a9d21681edfbbdd415fa78119264529c53c4c01b71d3c212d" gracePeriod=25
Nov 03 20:50:33.605248 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:33.603807    5591 generic.go:356] "Generic (PLEG): container finished" podID="4a78f787-3921-4287-a438-40ff5ebc7703" containerID="b98c36f07cc872ffd5f41aeb04cb2aa5d7cc34d216bf3070d0f4aacd51c7291e" exitCode=0
Nov 03 20:50:33.605248 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:33.603880    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-c44c86556-c65mk" event={"ID":"4a78f787-3921-4287-a438-40ff5ebc7703","Type":"ContainerDied","Data":"b98c36f07cc872ffd5f41aeb04cb2aa5d7cc34d216bf3070d0f4aacd51c7291e"}
Nov 03 20:50:33.747025 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:33.746649    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-c44c86556-c65mk"
Nov 03 20:50:33.821610 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:33.821564    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a78f787-3921-4287-a438-40ff5ebc7703-kube-api-access-kppl5\" (UniqueName: \"kubernetes.io/projected/4a78f787-3921-4287-a438-40ff5ebc7703-kube-api-access-kppl5\") pod \"4a78f787-3921-4287-a438-40ff5ebc7703\" (UID: \"4a78f787-3921-4287-a438-40ff5ebc7703\") "
Nov 03 20:50:33.821963 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:33.821944    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a78f787-3921-4287-a438-40ff5ebc7703-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a78f787-3921-4287-a438-40ff5ebc7703-plugins-config-volume\") pod \"4a78f787-3921-4287-a438-40ff5ebc7703\" (UID: \"4a78f787-3921-4287-a438-40ff5ebc7703\") "
Nov 03 20:50:33.822465 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:33.822415    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a78f787-3921-4287-a438-40ff5ebc7703-plugins-config-volume" pod "4a78f787-3921-4287-a438-40ff5ebc7703" (UID: "4a78f787-3921-4287-a438-40ff5ebc7703"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:50:33.822692 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:33.822672    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a78f787-3921-4287-a438-40ff5ebc7703-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:33.824643 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:33.824612    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a78f787-3921-4287-a438-40ff5ebc7703-kube-api-access-kppl5" pod "4a78f787-3921-4287-a438-40ff5ebc7703" (UID: "4a78f787-3921-4287-a438-40ff5ebc7703"). InnerVolumeSpecName "kube-api-access-kppl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:50:33.850960 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:33.850936    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-9c77b64f4-hdd5s"
Nov 03 20:50:33.923272 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:33.923224    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a2db403-e4b9-4e2b-ae5b-cf620bb89196-kube-api-access-k8wqj\" (UniqueName: \"kubernetes.io/projected/3a2db403-e4b9-4e2b-ae5b-cf620bb89196-kube-api-access-k8wqj\") pod \"3a2db403-e4b9-4e2b-ae5b-cf620bb89196\" (UID: \"3a2db403-e4b9-4e2b-ae5b-cf620bb89196\") "
Nov 03 20:50:33.923460 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:33.923310    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a2db403-e4b9-4e2b-ae5b-cf620bb89196-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a2db403-e4b9-4e2b-ae5b-cf620bb89196-plugins-config-volume\") pod \"3a2db403-e4b9-4e2b-ae5b-cf620bb89196\" (UID: \"3a2db403-e4b9-4e2b-ae5b-cf620bb89196\") "
Nov 03 20:50:33.923732 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:33.923706    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kppl5\" (UniqueName: \"kubernetes.io/projected/4a78f787-3921-4287-a438-40ff5ebc7703-kube-api-access-kppl5\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:33.923899 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:33.923792    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a2db403-e4b9-4e2b-ae5b-cf620bb89196-plugins-config-volume" pod "3a2db403-e4b9-4e2b-ae5b-cf620bb89196" (UID: "3a2db403-e4b9-4e2b-ae5b-cf620bb89196"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:50:33.925692 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:33.925667    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a2db403-e4b9-4e2b-ae5b-cf620bb89196-kube-api-access-k8wqj" pod "3a2db403-e4b9-4e2b-ae5b-cf620bb89196" (UID: "3a2db403-e4b9-4e2b-ae5b-cf620bb89196"). InnerVolumeSpecName "kube-api-access-k8wqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:50:34.039228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:34.038881    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a2db403-e4b9-4e2b-ae5b-cf620bb89196-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:34.039228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:34.038926    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8wqj\" (UniqueName: \"kubernetes.io/projected/3a2db403-e4b9-4e2b-ae5b-cf620bb89196-kube-api-access-k8wqj\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:50:34.638645 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:34.634752    5591 generic.go:356] "Generic (PLEG): container finished" podID="3a2db403-e4b9-4e2b-ae5b-cf620bb89196" containerID="caf142ac0e2fca0a9d21681edfbbdd415fa78119264529c53c4c01b71d3c212d" exitCode=0
Nov 03 20:50:34.638645 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:34.634849    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-9c77b64f4-hdd5s" event={"ID":"3a2db403-e4b9-4e2b-ae5b-cf620bb89196","Type":"ContainerDied","Data":"caf142ac0e2fca0a9d21681edfbbdd415fa78119264529c53c4c01b71d3c212d"}
Nov 03 20:50:34.638645 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:34.634905    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-9c77b64f4-hdd5s" event={"ID":"3a2db403-e4b9-4e2b-ae5b-cf620bb89196","Type":"ContainerDied","Data":"360fe592bd6889dc3ba16f8e63fb32f973520b02cc9da5cb1fa28d85cc3da6f3"}
Nov 03 20:50:34.638645 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:34.634937    5591 scope.go:122] "RemoveContainer" containerID="caf142ac0e2fca0a9d21681edfbbdd415fa78119264529c53c4c01b71d3c212d"
Nov 03 20:50:34.638645 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:34.635105    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-9c77b64f4-hdd5s"
Nov 03 20:50:34.654664 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:34.654631    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-c44c86556-c65mk" event={"ID":"4a78f787-3921-4287-a438-40ff5ebc7703","Type":"ContainerDied","Data":"2587bce07718fc271978759c927c4e436f806495cb064193538477c9a58a3445"}
Nov 03 20:50:34.655088 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:34.655068    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-c44c86556-c65mk"
Nov 03 20:50:34.666755 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:34.666732    5591 scope.go:122] "RemoveContainer" containerID="caf142ac0e2fca0a9d21681edfbbdd415fa78119264529c53c4c01b71d3c212d"
Nov 03 20:50:34.667048 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:50:34.667029    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caf142ac0e2fca0a9d21681edfbbdd415fa78119264529c53c4c01b71d3c212d\": container with ID starting with caf142ac0e2fca0a9d21681edfbbdd415fa78119264529c53c4c01b71d3c212d not found: ID does not exist" containerID="caf142ac0e2fca0a9d21681edfbbdd415fa78119264529c53c4c01b71d3c212d"
Nov 03 20:50:34.667145 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:34.667062    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caf142ac0e2fca0a9d21681edfbbdd415fa78119264529c53c4c01b71d3c212d"} err="failed to get container status \"caf142ac0e2fca0a9d21681edfbbdd415fa78119264529c53c4c01b71d3c212d\": rpc error: code = NotFound desc = could not find container \"caf142ac0e2fca0a9d21681edfbbdd415fa78119264529c53c4c01b71d3c212d\": container with ID starting with caf142ac0e2fca0a9d21681edfbbdd415fa78119264529c53c4c01b71d3c212d not found: ID does not exist"
Nov 03 20:50:34.667145 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:34.667085    5591 scope.go:122] "RemoveContainer" containerID="b98c36f07cc872ffd5f41aeb04cb2aa5d7cc34d216bf3070d0f4aacd51c7291e"
Nov 03 20:50:34.699279 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:34.687464    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-9c77b64f4-hdd5s"]
Nov 03 20:50:34.713349 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:34.713322    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-9c77b64f4-hdd5s"]
Nov 03 20:50:34.747610 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:34.747525    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-c44c86556-c65mk"]
Nov 03 20:50:34.749493 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:34.749471    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-c44c86556-c65mk"]
Nov 03 20:50:35.260419 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:35.260372    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-zst5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:50:35.260628 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:35.260447    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d" podUID="e8c03ef4-fb42-4a08-b02d-98165eb9b203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:50:35.352194 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:35.352149    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a2db403-e4b9-4e2b-ae5b-cf620bb89196" path="/var/lib/kubelet/pods/3a2db403-e4b9-4e2b-ae5b-cf620bb89196/volumes"
Nov 03 20:50:35.353019 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:35.352993    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a78f787-3921-4287-a438-40ff5ebc7703" path="/var/lib/kubelet/pods/4a78f787-3921-4287-a438-40ff5ebc7703/volumes"
Nov 03 20:50:36.023479 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:36.023441    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3953989c-x4ms7"]
Nov 03 20:50:36.026713 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:36.026688    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3953989c-x4ms7"]
Nov 03 20:50:37.347245 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:37.347195    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43c8ac25-77ff-4dc1-ae09-050cd7e4504a" path="/var/lib/kubelet/pods/43c8ac25-77ff-4dc1-ae09-050cd7e4504a/volumes"
Nov 03 20:50:43.607287 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:50:43.607248    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9"
Nov 03 20:50:58.724633 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:50:58.724589    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:51:02.037688 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:51:02.037648    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4da46e29-vm69x"]
Nov 03 20:51:02.040506 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:51:02.040480    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4da46e29-vm69x"]
Nov 03 20:51:03.349195 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:51:03.349160    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e06afc0a-1823-4700-b5be-42206eb9ecb8" path="/var/lib/kubelet/pods/e06afc0a-1823-4700-b5be-42206eb9ecb8/volumes"
Nov 03 20:51:28.724447 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:51:28.724400    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:51:38.166168 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:51:38.166133    5591 scope.go:122] "RemoveContainer" containerID="c7b5e7ca8af5dbc5d03d508aa7a06fbf71a53382779111f8d320b9eaedd68797"
Nov 03 20:51:58.724570 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:51:58.724527    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:52:07.003497 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:07.003462    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-2cnnn"]
Nov 03 20:52:07.004003 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:07.003890    5591 cpu_manager.go:414] "containerMap: removing container" podUID="3a2db403-e4b9-4e2b-ae5b-cf620bb89196" containerName="payload-processing"
Nov 03 20:52:07.004003 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:07.003901    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a2db403-e4b9-4e2b-ae5b-cf620bb89196" containerName="payload-processing"
Nov 03 20:52:07.004003 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:07.003939    5591 cpu_manager.go:414] "containerMap: removing container" podUID="4a78f787-3921-4287-a438-40ff5ebc7703" containerName="payload-pre-processing"
Nov 03 20:52:07.004003 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:07.003944    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a78f787-3921-4287-a438-40ff5ebc7703" containerName="payload-pre-processing"
Nov 03 20:52:07.004003 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:07.004005    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a2db403-e4b9-4e2b-ae5b-cf620bb89196" containerName="payload-processing"
Nov 03 20:52:07.004176 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:07.004021    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a78f787-3921-4287-a438-40ff5ebc7703" containerName="payload-pre-processing"
Nov 03 20:52:07.007053 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:07.007037    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-2cnnn"
Nov 03 20:52:07.014410 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:07.013827    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:52:07.017404 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:07.017361    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 20:52:07.017964 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:07.017413    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:52:07.018352 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:07.018328    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:52:07.022747 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:07.022415    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-2cnnn"]
Nov 03 20:52:07.037579 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:07.037545    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2aeeb83-68ec-44c3-84e7-91350fb5d744-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-2cnnn\" (UID: \"c2aeeb83-68ec-44c3-84e7-91350fb5d744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-2cnnn"
Nov 03 20:52:07.037731 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:07.037645    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2aeeb83-68ec-44c3-84e7-91350fb5d744-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-2cnnn\" (UID: \"c2aeeb83-68ec-44c3-84e7-91350fb5d744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-2cnnn"
Nov 03 20:52:07.037731 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:07.037689    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2kl6\" (UniqueName: \"kubernetes.io/projected/c2aeeb83-68ec-44c3-84e7-91350fb5d744-kube-api-access-m2kl6\") pod \"maas-api-revoke-sub-e2547622-2cnnn\" (UID: \"c2aeeb83-68ec-44c3-84e7-91350fb5d744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-2cnnn"
Nov 03 20:52:07.138678 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:07.138642    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2kl6\" (UniqueName: \"kubernetes.io/projected/c2aeeb83-68ec-44c3-84e7-91350fb5d744-kube-api-access-m2kl6\") pod \"maas-api-revoke-sub-e2547622-2cnnn\" (UID: \"c2aeeb83-68ec-44c3-84e7-91350fb5d744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-2cnnn"
Nov 03 20:52:07.138870 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:07.138746    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2aeeb83-68ec-44c3-84e7-91350fb5d744-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-2cnnn\" (UID: \"c2aeeb83-68ec-44c3-84e7-91350fb5d744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-2cnnn"
Nov 03 20:52:07.138870 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:07.138783    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2aeeb83-68ec-44c3-84e7-91350fb5d744-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-2cnnn\" (UID: \"c2aeeb83-68ec-44c3-84e7-91350fb5d744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-2cnnn"
Nov 03 20:52:07.139423 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:07.139400    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2aeeb83-68ec-44c3-84e7-91350fb5d744-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-2cnnn\" (UID: \"c2aeeb83-68ec-44c3-84e7-91350fb5d744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-2cnnn"
Nov 03 20:52:07.141339 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:07.141305    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2aeeb83-68ec-44c3-84e7-91350fb5d744-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-2cnnn\" (UID: \"c2aeeb83-68ec-44c3-84e7-91350fb5d744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-2cnnn"
Nov 03 20:52:07.151558 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:07.151533    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2kl6\" (UniqueName: \"kubernetes.io/projected/c2aeeb83-68ec-44c3-84e7-91350fb5d744-kube-api-access-m2kl6\") pod \"maas-api-revoke-sub-e2547622-2cnnn\" (UID: \"c2aeeb83-68ec-44c3-84e7-91350fb5d744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-2cnnn"
Nov 03 20:52:07.328242 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:07.328185    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-2cnnn"
Nov 03 20:52:07.609638 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:07.609515    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-2cnnn"]
Nov 03 20:52:07.612582 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:52:07.612549    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2aeeb83_68ec_44c3_84e7_91350fb5d744.slice/crio-ec5c774b4d9e46d85e5caa344c1592240239484035ddb6f9d797b7589fe52e15 WatchSource:0}: Error finding container ec5c774b4d9e46d85e5caa344c1592240239484035ddb6f9d797b7589fe52e15: Status 404 returned error can't find the container with id ec5c774b4d9e46d85e5caa344c1592240239484035ddb6f9d797b7589fe52e15
Nov 03 20:52:08.124472 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:08.124428    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-2cnnn" event={"ID":"c2aeeb83-68ec-44c3-84e7-91350fb5d744","Type":"ContainerStarted","Data":"00e8aa151a86a579b164803bfb5d9c3d395fcc085370b814bf38163d68866ef4"}
Nov 03 20:52:08.124900 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:08.124492    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-2cnnn" event={"ID":"c2aeeb83-68ec-44c3-84e7-91350fb5d744","Type":"ContainerStarted","Data":"ec5c774b4d9e46d85e5caa344c1592240239484035ddb6f9d797b7589fe52e15"}
Nov 03 20:52:08.144663 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:08.144560    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-2cnnn" podStartSLOduration=2.144544705 podStartE2EDuration="2.144544705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:08.142894806 +0000 UTC m=+4476.915187825" watchObservedRunningTime="2026-11-03 20:52:08.144544705 +0000 UTC m=+4476.916837901"
Nov 03 20:52:09.131090 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:09.131056    5591 generic.go:356] "Generic (PLEG): container finished" podID="c2aeeb83-68ec-44c3-84e7-91350fb5d744" containerID="00e8aa151a86a579b164803bfb5d9c3d395fcc085370b814bf38163d68866ef4" exitCode=0
Nov 03 20:52:09.131090 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:09.131094    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-2cnnn" event={"ID":"c2aeeb83-68ec-44c3-84e7-91350fb5d744","Type":"ContainerDied","Data":"00e8aa151a86a579b164803bfb5d9c3d395fcc085370b814bf38163d68866ef4"}
Nov 03 20:52:10.272871 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:10.272847    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-2cnnn"
Nov 03 20:52:10.364099 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:10.364059    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2aeeb83-68ec-44c3-84e7-91350fb5d744-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2aeeb83-68ec-44c3-84e7-91350fb5d744-maas-api-xks-ca\") pod \"c2aeeb83-68ec-44c3-84e7-91350fb5d744\" (UID: \"c2aeeb83-68ec-44c3-84e7-91350fb5d744\") "
Nov 03 20:52:10.364099 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:10.364106    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2aeeb83-68ec-44c3-84e7-91350fb5d744-kube-api-access-m2kl6\" (UniqueName: \"kubernetes.io/projected/c2aeeb83-68ec-44c3-84e7-91350fb5d744-kube-api-access-m2kl6\") pod \"c2aeeb83-68ec-44c3-84e7-91350fb5d744\" (UID: \"c2aeeb83-68ec-44c3-84e7-91350fb5d744\") "
Nov 03 20:52:10.364430 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:10.364164    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2aeeb83-68ec-44c3-84e7-91350fb5d744-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2aeeb83-68ec-44c3-84e7-91350fb5d744-maas-api-openshift-ca\") pod \"c2aeeb83-68ec-44c3-84e7-91350fb5d744\" (UID: \"c2aeeb83-68ec-44c3-84e7-91350fb5d744\") "
Nov 03 20:52:10.364677 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:10.364641    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2aeeb83-68ec-44c3-84e7-91350fb5d744-maas-api-openshift-ca" pod "c2aeeb83-68ec-44c3-84e7-91350fb5d744" (UID: "c2aeeb83-68ec-44c3-84e7-91350fb5d744"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:10.366481 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:10.366453    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2aeeb83-68ec-44c3-84e7-91350fb5d744-maas-api-xks-ca" pod "c2aeeb83-68ec-44c3-84e7-91350fb5d744" (UID: "c2aeeb83-68ec-44c3-84e7-91350fb5d744"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:52:10.366575 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:10.366535    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2aeeb83-68ec-44c3-84e7-91350fb5d744-kube-api-access-m2kl6" pod "c2aeeb83-68ec-44c3-84e7-91350fb5d744" (UID: "c2aeeb83-68ec-44c3-84e7-91350fb5d744"). InnerVolumeSpecName "kube-api-access-m2kl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:10.465177 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:10.465091    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2aeeb83-68ec-44c3-84e7-91350fb5d744-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:10.465177 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:10.465121    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2aeeb83-68ec-44c3-84e7-91350fb5d744-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:10.465177 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:10.465132    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2kl6\" (UniqueName: \"kubernetes.io/projected/c2aeeb83-68ec-44c3-84e7-91350fb5d744-kube-api-access-m2kl6\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:11.142455 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:11.142419    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-2cnnn"
Nov 03 20:52:11.142634 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:11.142471    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-2cnnn" event={"ID":"c2aeeb83-68ec-44c3-84e7-91350fb5d744","Type":"ContainerDied","Data":"ec5c774b4d9e46d85e5caa344c1592240239484035ddb6f9d797b7589fe52e15"}
Nov 03 20:52:11.142634 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:11.142527    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec5c774b4d9e46d85e5caa344c1592240239484035ddb6f9d797b7589fe52e15"
Nov 03 20:52:17.608285 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:17.608242    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9"]
Nov 03 20:52:18.036319 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:18.036279    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:52:22.542106 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:22.541317    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:52:22.611679 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:22.611611    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9" podUID="6c08bc5b-267a-4356-93c3-c2629a24a486" containerName="payload-processing" containerID="cri-o://b842a996fa7986b25dcc2c67f5d9b968868b52a68f9f23f93bb9b5f181543698" gracePeriod=25
Nov 03 20:52:23.000290 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:23.000189    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9"
Nov 03 20:52:23.138685 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:23.138570    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c08bc5b-267a-4356-93c3-c2629a24a486-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c08bc5b-267a-4356-93c3-c2629a24a486-routing-overlay-volume\") pod \"6c08bc5b-267a-4356-93c3-c2629a24a486\" (UID: \"6c08bc5b-267a-4356-93c3-c2629a24a486\") "
Nov 03 20:52:23.138879 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:23.138764    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c08bc5b-267a-4356-93c3-c2629a24a486-kube-api-access-p2rs9\" (UniqueName: \"kubernetes.io/projected/6c08bc5b-267a-4356-93c3-c2629a24a486-kube-api-access-p2rs9\") pod \"6c08bc5b-267a-4356-93c3-c2629a24a486\" (UID: \"6c08bc5b-267a-4356-93c3-c2629a24a486\") "
Nov 03 20:52:23.138879 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:23.138827    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c08bc5b-267a-4356-93c3-c2629a24a486-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c08bc5b-267a-4356-93c3-c2629a24a486-provider-credentials\") pod \"6c08bc5b-267a-4356-93c3-c2629a24a486\" (UID: \"6c08bc5b-267a-4356-93c3-c2629a24a486\") "
Nov 03 20:52:23.138879 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:23.138876    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c08bc5b-267a-4356-93c3-c2629a24a486-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c08bc5b-267a-4356-93c3-c2629a24a486-plugins-config-volume\") pod \"6c08bc5b-267a-4356-93c3-c2629a24a486\" (UID: \"6c08bc5b-267a-4356-93c3-c2629a24a486\") "
Nov 03 20:52:23.139492 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:23.139458    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c08bc5b-267a-4356-93c3-c2629a24a486-routing-overlay-volume" pod "6c08bc5b-267a-4356-93c3-c2629a24a486" (UID: "6c08bc5b-267a-4356-93c3-c2629a24a486"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:23.139640 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:23.139537    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c08bc5b-267a-4356-93c3-c2629a24a486-plugins-config-volume" pod "6c08bc5b-267a-4356-93c3-c2629a24a486" (UID: "6c08bc5b-267a-4356-93c3-c2629a24a486"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:23.142164 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:23.142139    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c08bc5b-267a-4356-93c3-c2629a24a486-kube-api-access-p2rs9" pod "6c08bc5b-267a-4356-93c3-c2629a24a486" (UID: "6c08bc5b-267a-4356-93c3-c2629a24a486"). InnerVolumeSpecName "kube-api-access-p2rs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:23.154334 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:23.154306    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c08bc5b-267a-4356-93c3-c2629a24a486-provider-credentials" pod "6c08bc5b-267a-4356-93c3-c2629a24a486" (UID: "6c08bc5b-267a-4356-93c3-c2629a24a486"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:23.234249 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:23.234178    5591 generic.go:356] "Generic (PLEG): container finished" podID="6c08bc5b-267a-4356-93c3-c2629a24a486" containerID="b842a996fa7986b25dcc2c67f5d9b968868b52a68f9f23f93bb9b5f181543698" exitCode=0
Nov 03 20:52:23.234440 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:23.234261    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9"
Nov 03 20:52:23.234440 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:23.234259    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9" event={"ID":"6c08bc5b-267a-4356-93c3-c2629a24a486","Type":"ContainerDied","Data":"b842a996fa7986b25dcc2c67f5d9b968868b52a68f9f23f93bb9b5f181543698"}
Nov 03 20:52:23.234440 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:23.234408    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9" event={"ID":"6c08bc5b-267a-4356-93c3-c2629a24a486","Type":"ContainerDied","Data":"8b97e48441b6d1fe0bd65bb911947fb472cc00a87ed559768203f29479790e66"}
Nov 03 20:52:23.234592 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:23.234451    5591 scope.go:122] "RemoveContainer" containerID="b842a996fa7986b25dcc2c67f5d9b968868b52a68f9f23f93bb9b5f181543698"
Nov 03 20:52:23.240657 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:23.240631    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c08bc5b-267a-4356-93c3-c2629a24a486-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:23.240854 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:23.240836    5591 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6c08bc5b-267a-4356-93c3-c2629a24a486-routing-overlay-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:23.240982 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:23.240955    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2rs9\" (UniqueName: \"kubernetes.io/projected/6c08bc5b-267a-4356-93c3-c2629a24a486-kube-api-access-p2rs9\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:23.241097 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:23.241084    5591 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6c08bc5b-267a-4356-93c3-c2629a24a486-provider-credentials\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:23.245303 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:23.245135    5591 scope.go:122] "RemoveContainer" containerID="b842a996fa7986b25dcc2c67f5d9b968868b52a68f9f23f93bb9b5f181543698"
Nov 03 20:52:23.245681 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:52:23.245525    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b842a996fa7986b25dcc2c67f5d9b968868b52a68f9f23f93bb9b5f181543698\": container with ID starting with b842a996fa7986b25dcc2c67f5d9b968868b52a68f9f23f93bb9b5f181543698 not found: ID does not exist" containerID="b842a996fa7986b25dcc2c67f5d9b968868b52a68f9f23f93bb9b5f181543698"
Nov 03 20:52:23.245681 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:23.245577    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b842a996fa7986b25dcc2c67f5d9b968868b52a68f9f23f93bb9b5f181543698"} err="failed to get container status \"b842a996fa7986b25dcc2c67f5d9b968868b52a68f9f23f93bb9b5f181543698\": rpc error: code = NotFound desc = could not find container \"b842a996fa7986b25dcc2c67f5d9b968868b52a68f9f23f93bb9b5f181543698\": container with ID starting with b842a996fa7986b25dcc2c67f5d9b968868b52a68f9f23f93bb9b5f181543698 not found: ID does not exist"
Nov 03 20:52:23.260255 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:23.260228    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9"]
Nov 03 20:52:23.266285 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:23.266250    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7fd6967f4c-t5nx9"]
Nov 03 20:52:23.349102 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:23.349062    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c08bc5b-267a-4356-93c3-c2629a24a486" path="/var/lib/kubelet/pods/6c08bc5b-267a-4356-93c3-c2629a24a486/volumes"
Nov 03 20:52:28.626273 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:28.626235    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlcf9"]
Nov 03 20:52:28.626792 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:28.626677    5591 cpu_manager.go:414] "containerMap: removing container" podUID="c2aeeb83-68ec-44c3-84e7-91350fb5d744" containerName="revoke-keys"
Nov 03 20:52:28.626792 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:28.626690    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2aeeb83-68ec-44c3-84e7-91350fb5d744" containerName="revoke-keys"
Nov 03 20:52:28.626792 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:28.626706    5591 cpu_manager.go:414] "containerMap: removing container" podUID="6c08bc5b-267a-4356-93c3-c2629a24a486" containerName="payload-processing"
Nov 03 20:52:28.626792 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:28.626712    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c08bc5b-267a-4356-93c3-c2629a24a486" containerName="payload-processing"
Nov 03 20:52:28.626792 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:28.626783    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c08bc5b-267a-4356-93c3-c2629a24a486" containerName="payload-processing"
Nov 03 20:52:28.626792 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:28.626799    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2aeeb83-68ec-44c3-84e7-91350fb5d744" containerName="revoke-keys"
Nov 03 20:52:28.631365 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:28.631343    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlcf9"
Nov 03 20:52:28.638589 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:28.638563    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:52:28.638788 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:28.638692    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:52:28.638788 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:28.638779    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 20:52:28.638927 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:28.638835    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:52:28.639807 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:28.639782    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlcf9"]
Nov 03 20:52:28.698988 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:28.698957    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65cdbcb2-4ccd-4c5c-a1de-8ee225bea762-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-nlcf9\" (UID: \"65cdbcb2-4ccd-4c5c-a1de-8ee225bea762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlcf9"
Nov 03 20:52:28.699233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:28.699016    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65cdbcb2-4ccd-4c5c-a1de-8ee225bea762-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-nlcf9\" (UID: \"65cdbcb2-4ccd-4c5c-a1de-8ee225bea762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlcf9"
Nov 03 20:52:28.699233 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:28.699136    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccdjz\" (UniqueName: \"kubernetes.io/projected/65cdbcb2-4ccd-4c5c-a1de-8ee225bea762-kube-api-access-ccdjz\") pod \"maas-api-revoke-sub-c1e271a9-nlcf9\" (UID: \"65cdbcb2-4ccd-4c5c-a1de-8ee225bea762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlcf9"
Nov 03 20:52:28.722284 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:52:28.722247    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:52:28.800582 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:28.800544    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65cdbcb2-4ccd-4c5c-a1de-8ee225bea762-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-nlcf9\" (UID: \"65cdbcb2-4ccd-4c5c-a1de-8ee225bea762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlcf9"
Nov 03 20:52:28.800802 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:28.800630    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65cdbcb2-4ccd-4c5c-a1de-8ee225bea762-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-nlcf9\" (UID: \"65cdbcb2-4ccd-4c5c-a1de-8ee225bea762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlcf9"
Nov 03 20:52:28.800802 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:28.800682    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccdjz\" (UniqueName: \"kubernetes.io/projected/65cdbcb2-4ccd-4c5c-a1de-8ee225bea762-kube-api-access-ccdjz\") pod \"maas-api-revoke-sub-c1e271a9-nlcf9\" (UID: \"65cdbcb2-4ccd-4c5c-a1de-8ee225bea762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlcf9"
Nov 03 20:52:28.801303 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:28.801277    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65cdbcb2-4ccd-4c5c-a1de-8ee225bea762-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-nlcf9\" (UID: \"65cdbcb2-4ccd-4c5c-a1de-8ee225bea762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlcf9"
Nov 03 20:52:28.803128 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:28.803098    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65cdbcb2-4ccd-4c5c-a1de-8ee225bea762-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-nlcf9\" (UID: \"65cdbcb2-4ccd-4c5c-a1de-8ee225bea762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlcf9"
Nov 03 20:52:28.813400 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:28.813377    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccdjz\" (UniqueName: \"kubernetes.io/projected/65cdbcb2-4ccd-4c5c-a1de-8ee225bea762-kube-api-access-ccdjz\") pod \"maas-api-revoke-sub-c1e271a9-nlcf9\" (UID: \"65cdbcb2-4ccd-4c5c-a1de-8ee225bea762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlcf9"
Nov 03 20:52:28.948294 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:28.948186    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlcf9"
Nov 03 20:52:29.223393 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:29.223367    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlcf9"]
Nov 03 20:52:29.226249 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:52:29.226197    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65cdbcb2_4ccd_4c5c_a1de_8ee225bea762.slice/crio-3e4d6a9633e7895c516409cfcefe46ade1c4fd0569155a055e39fc05dc9331ef WatchSource:0}: Error finding container 3e4d6a9633e7895c516409cfcefe46ade1c4fd0569155a055e39fc05dc9331ef: Status 404 returned error can't find the container with id 3e4d6a9633e7895c516409cfcefe46ade1c4fd0569155a055e39fc05dc9331ef
Nov 03 20:52:29.277494 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:29.277461    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlcf9" event={"ID":"65cdbcb2-4ccd-4c5c-a1de-8ee225bea762","Type":"ContainerStarted","Data":"3e4d6a9633e7895c516409cfcefe46ade1c4fd0569155a055e39fc05dc9331ef"}
Nov 03 20:52:30.284518 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:30.284483    5591 generic.go:356] "Generic (PLEG): container finished" podID="65cdbcb2-4ccd-4c5c-a1de-8ee225bea762" containerID="3773c302c7b5fb7b80ff9f047fecb2ba24a607405b325c6d896508a9d335f53b" exitCode=0
Nov 03 20:52:30.284930 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:30.284539    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlcf9" event={"ID":"65cdbcb2-4ccd-4c5c-a1de-8ee225bea762","Type":"ContainerDied","Data":"3773c302c7b5fb7b80ff9f047fecb2ba24a607405b325c6d896508a9d335f53b"}
Nov 03 20:52:31.424804 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:31.424779    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlcf9"
Nov 03 20:52:31.531626 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:31.531591    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65cdbcb2-4ccd-4c5c-a1de-8ee225bea762-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65cdbcb2-4ccd-4c5c-a1de-8ee225bea762-maas-api-openshift-ca\") pod \"65cdbcb2-4ccd-4c5c-a1de-8ee225bea762\" (UID: \"65cdbcb2-4ccd-4c5c-a1de-8ee225bea762\") "
Nov 03 20:52:31.531803 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:31.531660    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65cdbcb2-4ccd-4c5c-a1de-8ee225bea762-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65cdbcb2-4ccd-4c5c-a1de-8ee225bea762-maas-api-xks-ca\") pod \"65cdbcb2-4ccd-4c5c-a1de-8ee225bea762\" (UID: \"65cdbcb2-4ccd-4c5c-a1de-8ee225bea762\") "
Nov 03 20:52:31.531803 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:31.531681    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65cdbcb2-4ccd-4c5c-a1de-8ee225bea762-kube-api-access-ccdjz\" (UniqueName: \"kubernetes.io/projected/65cdbcb2-4ccd-4c5c-a1de-8ee225bea762-kube-api-access-ccdjz\") pod \"65cdbcb2-4ccd-4c5c-a1de-8ee225bea762\" (UID: \"65cdbcb2-4ccd-4c5c-a1de-8ee225bea762\") "
Nov 03 20:52:31.531956 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:31.531933    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65cdbcb2-4ccd-4c5c-a1de-8ee225bea762-maas-api-openshift-ca" pod "65cdbcb2-4ccd-4c5c-a1de-8ee225bea762" (UID: "65cdbcb2-4ccd-4c5c-a1de-8ee225bea762"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:31.533915 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:31.533883    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65cdbcb2-4ccd-4c5c-a1de-8ee225bea762-maas-api-xks-ca" pod "65cdbcb2-4ccd-4c5c-a1de-8ee225bea762" (UID: "65cdbcb2-4ccd-4c5c-a1de-8ee225bea762"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:52:31.534038 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:31.533917    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65cdbcb2-4ccd-4c5c-a1de-8ee225bea762-kube-api-access-ccdjz" pod "65cdbcb2-4ccd-4c5c-a1de-8ee225bea762" (UID: "65cdbcb2-4ccd-4c5c-a1de-8ee225bea762"). InnerVolumeSpecName "kube-api-access-ccdjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:31.632673 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:31.632586    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65cdbcb2-4ccd-4c5c-a1de-8ee225bea762-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:31.632673 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:31.632618    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65cdbcb2-4ccd-4c5c-a1de-8ee225bea762-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:31.632673 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:31.632629    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccdjz\" (UniqueName: \"kubernetes.io/projected/65cdbcb2-4ccd-4c5c-a1de-8ee225bea762-kube-api-access-ccdjz\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:32.296386 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:32.296355    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlcf9"
Nov 03 20:52:32.296571 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:32.296385    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlcf9" event={"ID":"65cdbcb2-4ccd-4c5c-a1de-8ee225bea762","Type":"ContainerDied","Data":"3e4d6a9633e7895c516409cfcefe46ade1c4fd0569155a055e39fc05dc9331ef"}
Nov 03 20:52:32.296571 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:32.296432    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e4d6a9633e7895c516409cfcefe46ade1c4fd0569155a055e39fc05dc9331ef"
Nov 03 20:52:34.221710 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:34.221677    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 20:52:34.229423 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:34.229401    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 20:52:38.191006 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:38.190974    5591 scope.go:122] "RemoveContainer" containerID="a98e498892de19189bdbea1f69ba06885df6c56f2e2b856c9a8b6f00b8dba97e"
Nov 03 20:52:39.508478 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:39.508433    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:52:50.800706 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:50.800610    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:52:55.184048 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:55.184007    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gk78w"]
Nov 03 20:52:55.184687 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:55.184665    5591 cpu_manager.go:414] "containerMap: removing container" podUID="65cdbcb2-4ccd-4c5c-a1de-8ee225bea762" containerName="revoke-keys"
Nov 03 20:52:55.184805 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:55.184689    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65cdbcb2-4ccd-4c5c-a1de-8ee225bea762" containerName="revoke-keys"
Nov 03 20:52:55.184805 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:55.184793    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="65cdbcb2-4ccd-4c5c-a1de-8ee225bea762" containerName="revoke-keys"
Nov 03 20:52:55.188169 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:55.188145    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gk78w"
Nov 03 20:52:55.191860 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:55.191826    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:52:55.191860 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:55.191831    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:52:55.192879 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:55.192859    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:52:55.194520 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:55.194500    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 20:52:55.198299 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:55.198276    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gk78w"]
Nov 03 20:52:55.275389 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:55.275355    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d5bf72-336a-4985-8906-5e2b8ae68345-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-gk78w\" (UID: \"31d5bf72-336a-4985-8906-5e2b8ae68345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gk78w"
Nov 03 20:52:55.275611 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:55.275421    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d5bf72-336a-4985-8906-5e2b8ae68345-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-gk78w\" (UID: \"31d5bf72-336a-4985-8906-5e2b8ae68345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gk78w"
Nov 03 20:52:55.275611 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:55.275522    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg5vf\" (UniqueName: \"kubernetes.io/projected/31d5bf72-336a-4985-8906-5e2b8ae68345-kube-api-access-bg5vf\") pod \"maas-api-revoke-sub-2e1f5b5f-gk78w\" (UID: \"31d5bf72-336a-4985-8906-5e2b8ae68345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gk78w"
Nov 03 20:52:55.376137 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:55.376086    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d5bf72-336a-4985-8906-5e2b8ae68345-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-gk78w\" (UID: \"31d5bf72-336a-4985-8906-5e2b8ae68345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gk78w"
Nov 03 20:52:55.376374 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:55.376177    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d5bf72-336a-4985-8906-5e2b8ae68345-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-gk78w\" (UID: \"31d5bf72-336a-4985-8906-5e2b8ae68345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gk78w"
Nov 03 20:52:55.376374 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:55.376238    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg5vf\" (UniqueName: \"kubernetes.io/projected/31d5bf72-336a-4985-8906-5e2b8ae68345-kube-api-access-bg5vf\") pod \"maas-api-revoke-sub-2e1f5b5f-gk78w\" (UID: \"31d5bf72-336a-4985-8906-5e2b8ae68345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gk78w"
Nov 03 20:52:55.376894 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:55.376873    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d5bf72-336a-4985-8906-5e2b8ae68345-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-gk78w\" (UID: \"31d5bf72-336a-4985-8906-5e2b8ae68345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gk78w"
Nov 03 20:52:55.378607 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:55.378581    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d5bf72-336a-4985-8906-5e2b8ae68345-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-gk78w\" (UID: \"31d5bf72-336a-4985-8906-5e2b8ae68345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gk78w"
Nov 03 20:52:55.389437 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:55.389413    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg5vf\" (UniqueName: \"kubernetes.io/projected/31d5bf72-336a-4985-8906-5e2b8ae68345-kube-api-access-bg5vf\") pod \"maas-api-revoke-sub-2e1f5b5f-gk78w\" (UID: \"31d5bf72-336a-4985-8906-5e2b8ae68345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gk78w"
Nov 03 20:52:55.503314 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:55.503227    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gk78w"
Nov 03 20:52:55.779987 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:55.779958    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gk78w"]
Nov 03 20:52:55.782225 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:52:55.782176    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d5bf72_336a_4985_8906_5e2b8ae68345.slice/crio-bdf3849bc6ad045f3887242ba0b1d84b0696144148ee4875a18c3c89ea48560a WatchSource:0}: Error finding container bdf3849bc6ad045f3887242ba0b1d84b0696144148ee4875a18c3c89ea48560a: Status 404 returned error can't find the container with id bdf3849bc6ad045f3887242ba0b1d84b0696144148ee4875a18c3c89ea48560a
Nov 03 20:52:56.464906 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:56.464803    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gk78w" event={"ID":"31d5bf72-336a-4985-8906-5e2b8ae68345","Type":"ContainerStarted","Data":"335e8f0ae55e58cddcfb2445a950e1e366969e1d11d48ada739f61cc52da41fd"}
Nov 03 20:52:56.464906 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:56.464865    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gk78w" event={"ID":"31d5bf72-336a-4985-8906-5e2b8ae68345","Type":"ContainerStarted","Data":"bdf3849bc6ad045f3887242ba0b1d84b0696144148ee4875a18c3c89ea48560a"}
Nov 03 20:52:56.485407 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:56.485357    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gk78w" podStartSLOduration=1.485341719 podStartE2EDuration="1.485341719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:56.483525004 +0000 UTC m=+4525.255818021" watchObservedRunningTime="2026-11-03 20:52:56.485341719 +0000 UTC m=+4525.257634738"
Nov 03 20:52:57.471674 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:57.471638    5591 generic.go:356] "Generic (PLEG): container finished" podID="31d5bf72-336a-4985-8906-5e2b8ae68345" containerID="335e8f0ae55e58cddcfb2445a950e1e366969e1d11d48ada739f61cc52da41fd" exitCode=0
Nov 03 20:52:57.472104 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:57.471730    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gk78w" event={"ID":"31d5bf72-336a-4985-8906-5e2b8ae68345","Type":"ContainerDied","Data":"335e8f0ae55e58cddcfb2445a950e1e366969e1d11d48ada739f61cc52da41fd"}
Nov 03 20:52:58.620946 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:58.620917    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gk78w"
Nov 03 20:52:58.708016 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:58.707986    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31d5bf72-336a-4985-8906-5e2b8ae68345-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d5bf72-336a-4985-8906-5e2b8ae68345-maas-api-openshift-ca\") pod \"31d5bf72-336a-4985-8906-5e2b8ae68345\" (UID: \"31d5bf72-336a-4985-8906-5e2b8ae68345\") "
Nov 03 20:52:58.708221 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:58.708139    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31d5bf72-336a-4985-8906-5e2b8ae68345-kube-api-access-bg5vf\" (UniqueName: \"kubernetes.io/projected/31d5bf72-336a-4985-8906-5e2b8ae68345-kube-api-access-bg5vf\") pod \"31d5bf72-336a-4985-8906-5e2b8ae68345\" (UID: \"31d5bf72-336a-4985-8906-5e2b8ae68345\") "
Nov 03 20:52:58.708221 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:58.708175    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31d5bf72-336a-4985-8906-5e2b8ae68345-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d5bf72-336a-4985-8906-5e2b8ae68345-maas-api-xks-ca\") pod \"31d5bf72-336a-4985-8906-5e2b8ae68345\" (UID: \"31d5bf72-336a-4985-8906-5e2b8ae68345\") "
Nov 03 20:52:58.708455 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:58.708430    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31d5bf72-336a-4985-8906-5e2b8ae68345-maas-api-openshift-ca" pod "31d5bf72-336a-4985-8906-5e2b8ae68345" (UID: "31d5bf72-336a-4985-8906-5e2b8ae68345"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:52:58.708587 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:58.708571    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d5bf72-336a-4985-8906-5e2b8ae68345-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:58.710319 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:58.710292    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31d5bf72-336a-4985-8906-5e2b8ae68345-maas-api-xks-ca" pod "31d5bf72-336a-4985-8906-5e2b8ae68345" (UID: "31d5bf72-336a-4985-8906-5e2b8ae68345"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:52:58.710564 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:58.710524    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d5bf72-336a-4985-8906-5e2b8ae68345-kube-api-access-bg5vf" pod "31d5bf72-336a-4985-8906-5e2b8ae68345" (UID: "31d5bf72-336a-4985-8906-5e2b8ae68345"). InnerVolumeSpecName "kube-api-access-bg5vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:52:58.728262 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:52:58.728144    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:52:58.809691 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:58.809642    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg5vf\" (UniqueName: \"kubernetes.io/projected/31d5bf72-336a-4985-8906-5e2b8ae68345-kube-api-access-bg5vf\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:58.809691 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:58.809687    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d5bf72-336a-4985-8906-5e2b8ae68345-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:52:59.483310 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:59.483278    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gk78w"
Nov 03 20:52:59.483508 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:59.483314    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gk78w" event={"ID":"31d5bf72-336a-4985-8906-5e2b8ae68345","Type":"ContainerDied","Data":"bdf3849bc6ad045f3887242ba0b1d84b0696144148ee4875a18c3c89ea48560a"}
Nov 03 20:52:59.483508 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:52:59.483361    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdf3849bc6ad045f3887242ba0b1d84b0696144148ee4875a18c3c89ea48560a"
Nov 03 20:53:06.046482 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:06.046443    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:53:06.154759 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:06.154725    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w6hg6"]
Nov 03 20:53:06.155517 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:06.155493    5591 cpu_manager.go:414] "containerMap: removing container" podUID="31d5bf72-336a-4985-8906-5e2b8ae68345" containerName="revoke-keys"
Nov 03 20:53:06.155581 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:06.155523    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31d5bf72-336a-4985-8906-5e2b8ae68345" containerName="revoke-keys"
Nov 03 20:53:06.155643 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:06.155630    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="31d5bf72-336a-4985-8906-5e2b8ae68345" containerName="revoke-keys"
Nov 03 20:53:06.159536 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:06.159515    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w6hg6"
Nov 03 20:53:06.164921 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:06.164893    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:53:06.165139 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:06.165013    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 20:53:06.167446 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:06.167420    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w6hg6"]
Nov 03 20:53:06.167601 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:06.167495    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:53:06.167601 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:06.167555    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:53:06.286088 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:06.286037    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/916f881b-1b9b-4be6-a507-8ac6313baf29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-w6hg6\" (UID: \"916f881b-1b9b-4be6-a507-8ac6313baf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w6hg6"
Nov 03 20:53:06.286360 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:06.286184    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/916f881b-1b9b-4be6-a507-8ac6313baf29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-w6hg6\" (UID: \"916f881b-1b9b-4be6-a507-8ac6313baf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w6hg6"
Nov 03 20:53:06.286360 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:06.286241    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxdlb\" (UniqueName: \"kubernetes.io/projected/916f881b-1b9b-4be6-a507-8ac6313baf29-kube-api-access-nxdlb\") pod \"maas-api-revoke-sub-ec843018-w6hg6\" (UID: \"916f881b-1b9b-4be6-a507-8ac6313baf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w6hg6"
Nov 03 20:53:06.387352 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:06.387252    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/916f881b-1b9b-4be6-a507-8ac6313baf29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-w6hg6\" (UID: \"916f881b-1b9b-4be6-a507-8ac6313baf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w6hg6"
Nov 03 20:53:06.387352 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:06.387322    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxdlb\" (UniqueName: \"kubernetes.io/projected/916f881b-1b9b-4be6-a507-8ac6313baf29-kube-api-access-nxdlb\") pod \"maas-api-revoke-sub-ec843018-w6hg6\" (UID: \"916f881b-1b9b-4be6-a507-8ac6313baf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w6hg6"
Nov 03 20:53:06.387618 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:06.387397    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/916f881b-1b9b-4be6-a507-8ac6313baf29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-w6hg6\" (UID: \"916f881b-1b9b-4be6-a507-8ac6313baf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w6hg6"
Nov 03 20:53:06.388053 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:06.388028    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/916f881b-1b9b-4be6-a507-8ac6313baf29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-w6hg6\" (UID: \"916f881b-1b9b-4be6-a507-8ac6313baf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w6hg6"
Nov 03 20:53:06.389829 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:06.389809    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/916f881b-1b9b-4be6-a507-8ac6313baf29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-w6hg6\" (UID: \"916f881b-1b9b-4be6-a507-8ac6313baf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w6hg6"
Nov 03 20:53:06.399879 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:06.399846    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxdlb\" (UniqueName: \"kubernetes.io/projected/916f881b-1b9b-4be6-a507-8ac6313baf29-kube-api-access-nxdlb\") pod \"maas-api-revoke-sub-ec843018-w6hg6\" (UID: \"916f881b-1b9b-4be6-a507-8ac6313baf29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w6hg6"
Nov 03 20:53:06.488619 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:06.488577    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w6hg6"
Nov 03 20:53:06.777702 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:06.777677    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w6hg6"]
Nov 03 20:53:06.780367 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:53:06.780331    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916f881b_1b9b_4be6_a507_8ac6313baf29.slice/crio-21b296be2215362b565de6f108bb92a00ca0ae9feb89c144945fc356696c6039 WatchSource:0}: Error finding container 21b296be2215362b565de6f108bb92a00ca0ae9feb89c144945fc356696c6039: Status 404 returned error can't find the container with id 21b296be2215362b565de6f108bb92a00ca0ae9feb89c144945fc356696c6039
Nov 03 20:53:07.529816 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:07.529779    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w6hg6" event={"ID":"916f881b-1b9b-4be6-a507-8ac6313baf29","Type":"ContainerStarted","Data":"d87ce0742efd9d18aa943b641c229509d01137f854351cf2402ef8237bec62d5"}
Nov 03 20:53:07.530249 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:07.529840    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w6hg6" event={"ID":"916f881b-1b9b-4be6-a507-8ac6313baf29","Type":"ContainerStarted","Data":"21b296be2215362b565de6f108bb92a00ca0ae9feb89c144945fc356696c6039"}
Nov 03 20:53:07.550241 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:07.550173    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w6hg6" podStartSLOduration=1.550159785 podStartE2EDuration="1.550159785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:07.548975471 +0000 UTC m=+4536.321268488" watchObservedRunningTime="2026-11-03 20:53:07.550159785 +0000 UTC m=+4536.322452797"
Nov 03 20:53:08.538020 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:08.537980    5591 generic.go:356] "Generic (PLEG): container finished" podID="916f881b-1b9b-4be6-a507-8ac6313baf29" containerID="d87ce0742efd9d18aa943b641c229509d01137f854351cf2402ef8237bec62d5" exitCode=0
Nov 03 20:53:08.538619 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:08.538080    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w6hg6" event={"ID":"916f881b-1b9b-4be6-a507-8ac6313baf29","Type":"ContainerDied","Data":"d87ce0742efd9d18aa943b641c229509d01137f854351cf2402ef8237bec62d5"}
Nov 03 20:53:10.053231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:10.050541    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w6hg6"
Nov 03 20:53:10.159386 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:10.158539    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/916f881b-1b9b-4be6-a507-8ac6313baf29-kube-api-access-nxdlb\" (UniqueName: \"kubernetes.io/projected/916f881b-1b9b-4be6-a507-8ac6313baf29-kube-api-access-nxdlb\") pod \"916f881b-1b9b-4be6-a507-8ac6313baf29\" (UID: \"916f881b-1b9b-4be6-a507-8ac6313baf29\") "
Nov 03 20:53:10.159386 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:10.158667    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/916f881b-1b9b-4be6-a507-8ac6313baf29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/916f881b-1b9b-4be6-a507-8ac6313baf29-maas-api-xks-ca\") pod \"916f881b-1b9b-4be6-a507-8ac6313baf29\" (UID: \"916f881b-1b9b-4be6-a507-8ac6313baf29\") "
Nov 03 20:53:10.159386 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:10.158738    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/916f881b-1b9b-4be6-a507-8ac6313baf29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/916f881b-1b9b-4be6-a507-8ac6313baf29-maas-api-openshift-ca\") pod \"916f881b-1b9b-4be6-a507-8ac6313baf29\" (UID: \"916f881b-1b9b-4be6-a507-8ac6313baf29\") "
Nov 03 20:53:10.160253 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:10.159899    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/916f881b-1b9b-4be6-a507-8ac6313baf29-maas-api-openshift-ca" pod "916f881b-1b9b-4be6-a507-8ac6313baf29" (UID: "916f881b-1b9b-4be6-a507-8ac6313baf29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:53:10.162421 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:10.162328    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916f881b-1b9b-4be6-a507-8ac6313baf29-kube-api-access-nxdlb" pod "916f881b-1b9b-4be6-a507-8ac6313baf29" (UID: "916f881b-1b9b-4be6-a507-8ac6313baf29"). InnerVolumeSpecName "kube-api-access-nxdlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:53:10.162421 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:10.162399    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/916f881b-1b9b-4be6-a507-8ac6313baf29-maas-api-xks-ca" pod "916f881b-1b9b-4be6-a507-8ac6313baf29" (UID: "916f881b-1b9b-4be6-a507-8ac6313baf29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:53:10.260496 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:10.260404    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/916f881b-1b9b-4be6-a507-8ac6313baf29-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:10.260496 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:10.260437    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxdlb\" (UniqueName: \"kubernetes.io/projected/916f881b-1b9b-4be6-a507-8ac6313baf29-kube-api-access-nxdlb\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:10.260496 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:10.260454    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/916f881b-1b9b-4be6-a507-8ac6313baf29-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:10.558781 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:10.558745    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w6hg6"
Nov 03 20:53:10.558977 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:10.558774    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w6hg6" event={"ID":"916f881b-1b9b-4be6-a507-8ac6313baf29","Type":"ContainerDied","Data":"21b296be2215362b565de6f108bb92a00ca0ae9feb89c144945fc356696c6039"}
Nov 03 20:53:10.558977 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:10.558833    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21b296be2215362b565de6f108bb92a00ca0ae9feb89c144945fc356696c6039"
Nov 03 20:53:18.914967 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:18.914931    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:53:28.724182 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:53:28.724138    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:53:29.506670 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:29.506634    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhjqf"]
Nov 03 20:53:29.507402 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:29.507377    5591 cpu_manager.go:414] "containerMap: removing container" podUID="916f881b-1b9b-4be6-a507-8ac6313baf29" containerName="revoke-keys"
Nov 03 20:53:29.507534 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:29.507403    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="916f881b-1b9b-4be6-a507-8ac6313baf29" containerName="revoke-keys"
Nov 03 20:53:29.507534 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:29.507516    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="916f881b-1b9b-4be6-a507-8ac6313baf29" containerName="revoke-keys"
Nov 03 20:53:29.510799 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:29.510779    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhjqf"
Nov 03 20:53:29.517068 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:29.517041    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:53:29.517416 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:29.517100    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:53:29.518881 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:29.518861    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 20:53:29.518988 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:29.518943    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:53:29.521175 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:29.521141    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhjqf"]
Nov 03 20:53:29.584067 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:29.584037    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f6e1af8-cfd9-45e6-82ac-3887872b8406-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-lhjqf\" (UID: \"6f6e1af8-cfd9-45e6-82ac-3887872b8406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhjqf"
Nov 03 20:53:29.584281 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:29.584104    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f6e1af8-cfd9-45e6-82ac-3887872b8406-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-lhjqf\" (UID: \"6f6e1af8-cfd9-45e6-82ac-3887872b8406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhjqf"
Nov 03 20:53:29.584281 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:29.584235    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lls5m\" (UniqueName: \"kubernetes.io/projected/6f6e1af8-cfd9-45e6-82ac-3887872b8406-kube-api-access-lls5m\") pod \"maas-api-revoke-sub-046e1343-lhjqf\" (UID: \"6f6e1af8-cfd9-45e6-82ac-3887872b8406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhjqf"
Nov 03 20:53:29.685777 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:29.685733    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lls5m\" (UniqueName: \"kubernetes.io/projected/6f6e1af8-cfd9-45e6-82ac-3887872b8406-kube-api-access-lls5m\") pod \"maas-api-revoke-sub-046e1343-lhjqf\" (UID: \"6f6e1af8-cfd9-45e6-82ac-3887872b8406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhjqf"
Nov 03 20:53:29.686002 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:29.685827    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f6e1af8-cfd9-45e6-82ac-3887872b8406-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-lhjqf\" (UID: \"6f6e1af8-cfd9-45e6-82ac-3887872b8406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhjqf"
Nov 03 20:53:29.686002 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:29.685898    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f6e1af8-cfd9-45e6-82ac-3887872b8406-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-lhjqf\" (UID: \"6f6e1af8-cfd9-45e6-82ac-3887872b8406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhjqf"
Nov 03 20:53:29.686603 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:29.686582    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f6e1af8-cfd9-45e6-82ac-3887872b8406-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-lhjqf\" (UID: \"6f6e1af8-cfd9-45e6-82ac-3887872b8406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhjqf"
Nov 03 20:53:29.688455 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:29.688429    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f6e1af8-cfd9-45e6-82ac-3887872b8406-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-lhjqf\" (UID: \"6f6e1af8-cfd9-45e6-82ac-3887872b8406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhjqf"
Nov 03 20:53:29.698488 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:29.698460    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lls5m\" (UniqueName: \"kubernetes.io/projected/6f6e1af8-cfd9-45e6-82ac-3887872b8406-kube-api-access-lls5m\") pod \"maas-api-revoke-sub-046e1343-lhjqf\" (UID: \"6f6e1af8-cfd9-45e6-82ac-3887872b8406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhjqf"
Nov 03 20:53:29.825240 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:29.825193    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhjqf"
Nov 03 20:53:30.101912 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:30.101884    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhjqf"]
Nov 03 20:53:30.104882 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:53:30.104857    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f6e1af8_cfd9_45e6_82ac_3887872b8406.slice/crio-6b0701f1669946121ef83a6e324b402139091142eca43f9f71f22107cce03e24 WatchSource:0}: Error finding container 6b0701f1669946121ef83a6e324b402139091142eca43f9f71f22107cce03e24: Status 404 returned error can't find the container with id 6b0701f1669946121ef83a6e324b402139091142eca43f9f71f22107cce03e24
Nov 03 20:53:30.707582 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:30.707492    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhjqf" event={"ID":"6f6e1af8-cfd9-45e6-82ac-3887872b8406","Type":"ContainerStarted","Data":"94433728f80269c8cff785a8fcf1ea81910608e39b850cbdbe94915004b14afa"}
Nov 03 20:53:30.707582 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:30.707546    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhjqf" event={"ID":"6f6e1af8-cfd9-45e6-82ac-3887872b8406","Type":"ContainerStarted","Data":"6b0701f1669946121ef83a6e324b402139091142eca43f9f71f22107cce03e24"}
Nov 03 20:53:30.726979 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:30.726921    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhjqf" podStartSLOduration=1.7269064090000001 podStartE2EDuration="1.726906409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:30.725234525 +0000 UTC m=+4559.497527546" watchObservedRunningTime="2026-11-03 20:53:30.726906409 +0000 UTC m=+4559.499199438"
Nov 03 20:53:31.714321 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:31.714285    5591 generic.go:356] "Generic (PLEG): container finished" podID="6f6e1af8-cfd9-45e6-82ac-3887872b8406" containerID="94433728f80269c8cff785a8fcf1ea81910608e39b850cbdbe94915004b14afa" exitCode=0
Nov 03 20:53:31.714752 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:31.714378    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhjqf" event={"ID":"6f6e1af8-cfd9-45e6-82ac-3887872b8406","Type":"ContainerDied","Data":"94433728f80269c8cff785a8fcf1ea81910608e39b850cbdbe94915004b14afa"}
Nov 03 20:53:32.860014 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:32.859990    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhjqf"
Nov 03 20:53:33.020032 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:33.019999    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f6e1af8-cfd9-45e6-82ac-3887872b8406-kube-api-access-lls5m\" (UniqueName: \"kubernetes.io/projected/6f6e1af8-cfd9-45e6-82ac-3887872b8406-kube-api-access-lls5m\") pod \"6f6e1af8-cfd9-45e6-82ac-3887872b8406\" (UID: \"6f6e1af8-cfd9-45e6-82ac-3887872b8406\") "
Nov 03 20:53:33.020281 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:33.020070    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f6e1af8-cfd9-45e6-82ac-3887872b8406-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f6e1af8-cfd9-45e6-82ac-3887872b8406-maas-api-openshift-ca\") pod \"6f6e1af8-cfd9-45e6-82ac-3887872b8406\" (UID: \"6f6e1af8-cfd9-45e6-82ac-3887872b8406\") "
Nov 03 20:53:33.020281 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:33.020140    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f6e1af8-cfd9-45e6-82ac-3887872b8406-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f6e1af8-cfd9-45e6-82ac-3887872b8406-maas-api-xks-ca\") pod \"6f6e1af8-cfd9-45e6-82ac-3887872b8406\" (UID: \"6f6e1af8-cfd9-45e6-82ac-3887872b8406\") "
Nov 03 20:53:33.020541 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:33.020517    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f6e1af8-cfd9-45e6-82ac-3887872b8406-maas-api-openshift-ca" pod "6f6e1af8-cfd9-45e6-82ac-3887872b8406" (UID: "6f6e1af8-cfd9-45e6-82ac-3887872b8406"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:53:33.022457 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:33.022431    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f6e1af8-cfd9-45e6-82ac-3887872b8406-kube-api-access-lls5m" pod "6f6e1af8-cfd9-45e6-82ac-3887872b8406" (UID: "6f6e1af8-cfd9-45e6-82ac-3887872b8406"). InnerVolumeSpecName "kube-api-access-lls5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:53:33.022457 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:33.022446    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f6e1af8-cfd9-45e6-82ac-3887872b8406-maas-api-xks-ca" pod "6f6e1af8-cfd9-45e6-82ac-3887872b8406" (UID: "6f6e1af8-cfd9-45e6-82ac-3887872b8406"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:53:33.121431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:33.121396    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lls5m\" (UniqueName: \"kubernetes.io/projected/6f6e1af8-cfd9-45e6-82ac-3887872b8406-kube-api-access-lls5m\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:33.121431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:33.121425    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f6e1af8-cfd9-45e6-82ac-3887872b8406-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:33.121431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:33.121440    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f6e1af8-cfd9-45e6-82ac-3887872b8406-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:33.725455 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:33.725362    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhjqf" event={"ID":"6f6e1af8-cfd9-45e6-82ac-3887872b8406","Type":"ContainerDied","Data":"6b0701f1669946121ef83a6e324b402139091142eca43f9f71f22107cce03e24"}
Nov 03 20:53:33.725455 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:33.725398    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhjqf"
Nov 03 20:53:33.725455 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:33.725423    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b0701f1669946121ef83a6e324b402139091142eca43f9f71f22107cce03e24"
Nov 03 20:53:39.719247 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:39.719187    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-89rl2"]
Nov 03 20:53:39.719675 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:39.719661    5591 cpu_manager.go:414] "containerMap: removing container" podUID="6f6e1af8-cfd9-45e6-82ac-3887872b8406" containerName="revoke-keys"
Nov 03 20:53:39.719731 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:39.719676    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f6e1af8-cfd9-45e6-82ac-3887872b8406" containerName="revoke-keys"
Nov 03 20:53:39.719775 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:39.719738    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f6e1af8-cfd9-45e6-82ac-3887872b8406" containerName="revoke-keys"
Nov 03 20:53:39.723296 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:39.723274    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-89rl2"
Nov 03 20:53:39.730197 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:39.730167    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:53:39.732466 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:39.732314    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 20:53:39.732466 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:39.732407    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:53:39.733362 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:39.732590    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:53:39.733588 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:39.733558    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-89rl2"]
Nov 03 20:53:39.888714 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:39.888671    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033b1579-4ddf-4680-8c72-6bd296480c9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-89rl2\" (UID: \"033b1579-4ddf-4680-8c72-6bd296480c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-89rl2"
Nov 03 20:53:39.888921 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:39.888842    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnhhx\" (UniqueName: \"kubernetes.io/projected/033b1579-4ddf-4680-8c72-6bd296480c9e-kube-api-access-qnhhx\") pod \"maas-api-revoke-sub-d3fba056-89rl2\" (UID: \"033b1579-4ddf-4680-8c72-6bd296480c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-89rl2"
Nov 03 20:53:39.888921 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:39.888877    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033b1579-4ddf-4680-8c72-6bd296480c9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-89rl2\" (UID: \"033b1579-4ddf-4680-8c72-6bd296480c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-89rl2"
Nov 03 20:53:39.990402 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:39.990286    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnhhx\" (UniqueName: \"kubernetes.io/projected/033b1579-4ddf-4680-8c72-6bd296480c9e-kube-api-access-qnhhx\") pod \"maas-api-revoke-sub-d3fba056-89rl2\" (UID: \"033b1579-4ddf-4680-8c72-6bd296480c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-89rl2"
Nov 03 20:53:39.990402 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:39.990342    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033b1579-4ddf-4680-8c72-6bd296480c9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-89rl2\" (UID: \"033b1579-4ddf-4680-8c72-6bd296480c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-89rl2"
Nov 03 20:53:39.990402 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:39.990401    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033b1579-4ddf-4680-8c72-6bd296480c9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-89rl2\" (UID: \"033b1579-4ddf-4680-8c72-6bd296480c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-89rl2"
Nov 03 20:53:39.991142 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:39.991120    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033b1579-4ddf-4680-8c72-6bd296480c9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-89rl2\" (UID: \"033b1579-4ddf-4680-8c72-6bd296480c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-89rl2"
Nov 03 20:53:39.993015 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:39.992996    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033b1579-4ddf-4680-8c72-6bd296480c9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-89rl2\" (UID: \"033b1579-4ddf-4680-8c72-6bd296480c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-89rl2"
Nov 03 20:53:40.003095 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:40.003070    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnhhx\" (UniqueName: \"kubernetes.io/projected/033b1579-4ddf-4680-8c72-6bd296480c9e-kube-api-access-qnhhx\") pod \"maas-api-revoke-sub-d3fba056-89rl2\" (UID: \"033b1579-4ddf-4680-8c72-6bd296480c9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-89rl2"
Nov 03 20:53:40.040994 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:40.040952    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-89rl2"
Nov 03 20:53:40.320888 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:40.320860    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-89rl2"]
Nov 03 20:53:40.323375 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:53:40.323340    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod033b1579_4ddf_4680_8c72_6bd296480c9e.slice/crio-a6fa7b2b48c10b516779bac3c70c40a9a87d8c1cc613f2be2dda3efe47cca168 WatchSource:0}: Error finding container a6fa7b2b48c10b516779bac3c70c40a9a87d8c1cc613f2be2dda3efe47cca168: Status 404 returned error can't find the container with id a6fa7b2b48c10b516779bac3c70c40a9a87d8c1cc613f2be2dda3efe47cca168
Nov 03 20:53:40.437497 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:40.437459    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:53:40.762389 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:40.762347    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-89rl2" event={"ID":"033b1579-4ddf-4680-8c72-6bd296480c9e","Type":"ContainerStarted","Data":"dd877f497afd282d48d03f3402f6bc5fb73797d62dde81ed29363eaa5b1b2e4a"}
Nov 03 20:53:40.762988 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:40.762409    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-89rl2" event={"ID":"033b1579-4ddf-4680-8c72-6bd296480c9e","Type":"ContainerStarted","Data":"a6fa7b2b48c10b516779bac3c70c40a9a87d8c1cc613f2be2dda3efe47cca168"}
Nov 03 20:53:40.782101 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:40.782055    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-89rl2" podStartSLOduration=1.782041098 podStartE2EDuration="1.782041098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:40.779840722 +0000 UTC m=+4569.552133742" watchObservedRunningTime="2026-11-03 20:53:40.782041098 +0000 UTC m=+4569.554334116"
Nov 03 20:53:41.768752 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:41.768715    5591 generic.go:356] "Generic (PLEG): container finished" podID="033b1579-4ddf-4680-8c72-6bd296480c9e" containerID="dd877f497afd282d48d03f3402f6bc5fb73797d62dde81ed29363eaa5b1b2e4a" exitCode=0
Nov 03 20:53:41.769192 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:41.768804    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-89rl2" event={"ID":"033b1579-4ddf-4680-8c72-6bd296480c9e","Type":"ContainerDied","Data":"dd877f497afd282d48d03f3402f6bc5fb73797d62dde81ed29363eaa5b1b2e4a"}
Nov 03 20:53:43.171921 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:43.171893    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-89rl2"
Nov 03 20:53:43.355281 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:43.355244    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/033b1579-4ddf-4680-8c72-6bd296480c9e-kube-api-access-qnhhx\" (UniqueName: \"kubernetes.io/projected/033b1579-4ddf-4680-8c72-6bd296480c9e-kube-api-access-qnhhx\") pod \"033b1579-4ddf-4680-8c72-6bd296480c9e\" (UID: \"033b1579-4ddf-4680-8c72-6bd296480c9e\") "
Nov 03 20:53:43.355493 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:43.355345    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/033b1579-4ddf-4680-8c72-6bd296480c9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033b1579-4ddf-4680-8c72-6bd296480c9e-maas-api-xks-ca\") pod \"033b1579-4ddf-4680-8c72-6bd296480c9e\" (UID: \"033b1579-4ddf-4680-8c72-6bd296480c9e\") "
Nov 03 20:53:43.355493 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:43.355456    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/033b1579-4ddf-4680-8c72-6bd296480c9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033b1579-4ddf-4680-8c72-6bd296480c9e-maas-api-openshift-ca\") pod \"033b1579-4ddf-4680-8c72-6bd296480c9e\" (UID: \"033b1579-4ddf-4680-8c72-6bd296480c9e\") "
Nov 03 20:53:43.355889 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:43.355857    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/033b1579-4ddf-4680-8c72-6bd296480c9e-maas-api-openshift-ca" pod "033b1579-4ddf-4680-8c72-6bd296480c9e" (UID: "033b1579-4ddf-4680-8c72-6bd296480c9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:53:43.358255 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:43.358195    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033b1579-4ddf-4680-8c72-6bd296480c9e-kube-api-access-qnhhx" pod "033b1579-4ddf-4680-8c72-6bd296480c9e" (UID: "033b1579-4ddf-4680-8c72-6bd296480c9e"). InnerVolumeSpecName "kube-api-access-qnhhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:53:43.364347 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:43.364317    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/033b1579-4ddf-4680-8c72-6bd296480c9e-maas-api-xks-ca" pod "033b1579-4ddf-4680-8c72-6bd296480c9e" (UID: "033b1579-4ddf-4680-8c72-6bd296480c9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:53:43.457106 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:43.457006    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/033b1579-4ddf-4680-8c72-6bd296480c9e-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:43.457400 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:43.457367    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/033b1579-4ddf-4680-8c72-6bd296480c9e-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:43.457535 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:43.457403    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnhhx\" (UniqueName: \"kubernetes.io/projected/033b1579-4ddf-4680-8c72-6bd296480c9e-kube-api-access-qnhhx\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:53:43.780782 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:43.780737    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-89rl2" event={"ID":"033b1579-4ddf-4680-8c72-6bd296480c9e","Type":"ContainerDied","Data":"a6fa7b2b48c10b516779bac3c70c40a9a87d8c1cc613f2be2dda3efe47cca168"}
Nov 03 20:53:43.780970 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:43.780802    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6fa7b2b48c10b516779bac3c70c40a9a87d8c1cc613f2be2dda3efe47cca168"
Nov 03 20:53:43.780970 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:43.780749    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-89rl2"
Nov 03 20:53:50.544067 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:53:50.544028    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:53:58.725369 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:53:58.725329    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:54:02.318830 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:02.318790    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:54:05.241960 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:05.241918    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-zst5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:54:05.242523 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:05.242008    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zst5d" podUID="e8c03ef4-fb42-4a08-b02d-98165eb9b203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:54:07.026768 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:07.026721    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cbx9w"]
Nov 03 20:54:07.030893 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:07.030848    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-cbx9w"]
Nov 03 20:54:07.349728 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:07.349642    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7cea823-f76b-4e8c-a730-73da97dada6f" path="/var/lib/kubelet/pods/c7cea823-f76b-4e8c-a730-73da97dada6f/volumes"
Nov 03 20:54:07.959575 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:07.959537    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5b62q"]
Nov 03 20:54:07.959964 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:07.959953    5591 cpu_manager.go:414] "containerMap: removing container" podUID="033b1579-4ddf-4680-8c72-6bd296480c9e" containerName="revoke-keys"
Nov 03 20:54:07.960013 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:07.959966    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="033b1579-4ddf-4680-8c72-6bd296480c9e" containerName="revoke-keys"
Nov 03 20:54:07.960093 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:07.960084    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="033b1579-4ddf-4680-8c72-6bd296480c9e" containerName="revoke-keys"
Nov 03 20:54:07.964540 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:07.964518    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5b62q"
Nov 03 20:54:07.971072 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:07.971043    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:54:07.971375 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:07.971350    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:54:07.971375 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:07.971233    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 20:54:07.971584 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:07.971292    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:54:07.973069 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:07.973035    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5b62q"]
Nov 03 20:54:08.002130 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:08.002085    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c8a1c63-e110-435b-9b93-f6417a37d003-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-5b62q\" (UID: \"5c8a1c63-e110-435b-9b93-f6417a37d003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5b62q"
Nov 03 20:54:08.002321 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:08.002290    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hsw9\" (UniqueName: \"kubernetes.io/projected/5c8a1c63-e110-435b-9b93-f6417a37d003-kube-api-access-7hsw9\") pod \"maas-api-revoke-sub-ae594979-5b62q\" (UID: \"5c8a1c63-e110-435b-9b93-f6417a37d003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5b62q"
Nov 03 20:54:08.002374 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:08.002344    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c8a1c63-e110-435b-9b93-f6417a37d003-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-5b62q\" (UID: \"5c8a1c63-e110-435b-9b93-f6417a37d003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5b62q"
Nov 03 20:54:08.102867 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:08.102821    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hsw9\" (UniqueName: \"kubernetes.io/projected/5c8a1c63-e110-435b-9b93-f6417a37d003-kube-api-access-7hsw9\") pod \"maas-api-revoke-sub-ae594979-5b62q\" (UID: \"5c8a1c63-e110-435b-9b93-f6417a37d003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5b62q"
Nov 03 20:54:08.103300 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:08.102879    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c8a1c63-e110-435b-9b93-f6417a37d003-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-5b62q\" (UID: \"5c8a1c63-e110-435b-9b93-f6417a37d003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5b62q"
Nov 03 20:54:08.103300 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:08.102906    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c8a1c63-e110-435b-9b93-f6417a37d003-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-5b62q\" (UID: \"5c8a1c63-e110-435b-9b93-f6417a37d003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5b62q"
Nov 03 20:54:08.103586 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:08.103564    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c8a1c63-e110-435b-9b93-f6417a37d003-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-5b62q\" (UID: \"5c8a1c63-e110-435b-9b93-f6417a37d003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5b62q"
Nov 03 20:54:08.105550 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:08.105527    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c8a1c63-e110-435b-9b93-f6417a37d003-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-5b62q\" (UID: \"5c8a1c63-e110-435b-9b93-f6417a37d003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5b62q"
Nov 03 20:54:08.115608 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:08.115579    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hsw9\" (UniqueName: \"kubernetes.io/projected/5c8a1c63-e110-435b-9b93-f6417a37d003-kube-api-access-7hsw9\") pod \"maas-api-revoke-sub-ae594979-5b62q\" (UID: \"5c8a1c63-e110-435b-9b93-f6417a37d003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5b62q"
Nov 03 20:54:08.279275 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:08.279228    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5b62q"
Nov 03 20:54:08.561075 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:08.560997    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5b62q"]
Nov 03 20:54:08.564388 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:54:08.564350    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8a1c63_e110_435b_9b93_f6417a37d003.slice/crio-2ab29268c5de65cb38d4b8f2d7fcb9bb0ae36250c70cfecfd6847c2e7e60c23f WatchSource:0}: Error finding container 2ab29268c5de65cb38d4b8f2d7fcb9bb0ae36250c70cfecfd6847c2e7e60c23f: Status 404 returned error can't find the container with id 2ab29268c5de65cb38d4b8f2d7fcb9bb0ae36250c70cfecfd6847c2e7e60c23f
Nov 03 20:54:08.936711 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:08.936673    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5b62q" event={"ID":"5c8a1c63-e110-435b-9b93-f6417a37d003","Type":"ContainerStarted","Data":"573dca7a306adbfe5bb340c5684cedb8ec897c4e331ce69f130da54674cffd61"}
Nov 03 20:54:08.936911 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:08.936727    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5b62q" event={"ID":"5c8a1c63-e110-435b-9b93-f6417a37d003","Type":"ContainerStarted","Data":"2ab29268c5de65cb38d4b8f2d7fcb9bb0ae36250c70cfecfd6847c2e7e60c23f"}
Nov 03 20:54:08.955400 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:08.955340    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5b62q" podStartSLOduration=1.955320057 podStartE2EDuration="1.955320057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:08.955034059 +0000 UTC m=+4597.727327089" watchObservedRunningTime="2026-11-03 20:54:08.955320057 +0000 UTC m=+4597.727613080"
Nov 03 20:54:09.943387 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:09.943345    5591 generic.go:356] "Generic (PLEG): container finished" podID="5c8a1c63-e110-435b-9b93-f6417a37d003" containerID="573dca7a306adbfe5bb340c5684cedb8ec897c4e331ce69f130da54674cffd61" exitCode=0
Nov 03 20:54:09.943816 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:09.943436    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5b62q" event={"ID":"5c8a1c63-e110-435b-9b93-f6417a37d003","Type":"ContainerDied","Data":"573dca7a306adbfe5bb340c5684cedb8ec897c4e331ce69f130da54674cffd61"}
Nov 03 20:54:11.084793 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:11.084757    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5b62q"
Nov 03 20:54:11.134342 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:11.134311    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c8a1c63-e110-435b-9b93-f6417a37d003-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c8a1c63-e110-435b-9b93-f6417a37d003-maas-api-openshift-ca\") pod \"5c8a1c63-e110-435b-9b93-f6417a37d003\" (UID: \"5c8a1c63-e110-435b-9b93-f6417a37d003\") "
Nov 03 20:54:11.134534 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:11.134350    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c8a1c63-e110-435b-9b93-f6417a37d003-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c8a1c63-e110-435b-9b93-f6417a37d003-maas-api-xks-ca\") pod \"5c8a1c63-e110-435b-9b93-f6417a37d003\" (UID: \"5c8a1c63-e110-435b-9b93-f6417a37d003\") "
Nov 03 20:54:11.134534 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:11.134408    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c8a1c63-e110-435b-9b93-f6417a37d003-kube-api-access-7hsw9\" (UniqueName: \"kubernetes.io/projected/5c8a1c63-e110-435b-9b93-f6417a37d003-kube-api-access-7hsw9\") pod \"5c8a1c63-e110-435b-9b93-f6417a37d003\" (UID: \"5c8a1c63-e110-435b-9b93-f6417a37d003\") "
Nov 03 20:54:11.134685 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:11.134660    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c8a1c63-e110-435b-9b93-f6417a37d003-maas-api-openshift-ca" pod "5c8a1c63-e110-435b-9b93-f6417a37d003" (UID: "5c8a1c63-e110-435b-9b93-f6417a37d003"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:54:11.134826 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:11.134805    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c8a1c63-e110-435b-9b93-f6417a37d003-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:54:11.136710 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:11.136685    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c8a1c63-e110-435b-9b93-f6417a37d003-kube-api-access-7hsw9" pod "5c8a1c63-e110-435b-9b93-f6417a37d003" (UID: "5c8a1c63-e110-435b-9b93-f6417a37d003"). InnerVolumeSpecName "kube-api-access-7hsw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:54:11.136804 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:11.136708    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c8a1c63-e110-435b-9b93-f6417a37d003-maas-api-xks-ca" pod "5c8a1c63-e110-435b-9b93-f6417a37d003" (UID: "5c8a1c63-e110-435b-9b93-f6417a37d003"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:54:11.235623 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:11.235542    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hsw9\" (UniqueName: \"kubernetes.io/projected/5c8a1c63-e110-435b-9b93-f6417a37d003-kube-api-access-7hsw9\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:54:11.235623 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:11.235574    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c8a1c63-e110-435b-9b93-f6417a37d003-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:54:11.955824 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:11.955790    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5b62q"
Nov 03 20:54:11.955824 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:11.955815    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5b62q" event={"ID":"5c8a1c63-e110-435b-9b93-f6417a37d003","Type":"ContainerDied","Data":"2ab29268c5de65cb38d4b8f2d7fcb9bb0ae36250c70cfecfd6847c2e7e60c23f"}
Nov 03 20:54:11.956143 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:11.955863    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ab29268c5de65cb38d4b8f2d7fcb9bb0ae36250c70cfecfd6847c2e7e60c23f"
Nov 03 20:54:18.812414 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:18.812369    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:54:22.043981 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:22.043938    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nhsds"]
Nov 03 20:54:22.057235 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:22.051154    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-nhsds"]
Nov 03 20:54:22.628094 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:22.615142    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:54:23.353870 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:23.353831    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6852b9b4-89f9-4d67-abf4-a8d8042f1269" path="/var/lib/kubelet/pods/6852b9b4-89f9-4d67-abf4-a8d8042f1269/volumes"
Nov 03 20:54:24.613255 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:24.613221    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5wng8"]
Nov 03 20:54:24.613857 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:24.613835    5591 cpu_manager.go:414] "containerMap: removing container" podUID="5c8a1c63-e110-435b-9b93-f6417a37d003" containerName="revoke-keys"
Nov 03 20:54:24.613953 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:24.613859    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c8a1c63-e110-435b-9b93-f6417a37d003" containerName="revoke-keys"
Nov 03 20:54:24.614027 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:24.613989    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c8a1c63-e110-435b-9b93-f6417a37d003" containerName="revoke-keys"
Nov 03 20:54:24.617480 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:24.617458    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5wng8"
Nov 03 20:54:24.625366 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:24.625327    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:54:24.625510 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:24.625412    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 20:54:24.625992 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:24.625965    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:54:24.626131 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:24.626049    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:54:24.628152 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:24.628131    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5wng8"]
Nov 03 20:54:24.672663 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:24.672634    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-5wng8\" (UID: \"7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5wng8"
Nov 03 20:54:24.672852 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:24.672747    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-5wng8\" (UID: \"7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5wng8"
Nov 03 20:54:24.672852 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:24.672814    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t22n8\" (UniqueName: \"kubernetes.io/projected/7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1-kube-api-access-t22n8\") pod \"maas-api-revoke-sub-a4aef7d7-5wng8\" (UID: \"7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5wng8"
Nov 03 20:54:24.773444 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:24.773413    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-5wng8\" (UID: \"7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5wng8"
Nov 03 20:54:24.773647 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:24.773467    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t22n8\" (UniqueName: \"kubernetes.io/projected/7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1-kube-api-access-t22n8\") pod \"maas-api-revoke-sub-a4aef7d7-5wng8\" (UID: \"7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5wng8"
Nov 03 20:54:24.773647 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:24.773504    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-5wng8\" (UID: \"7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5wng8"
Nov 03 20:54:24.774056 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:24.774031    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-5wng8\" (UID: \"7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5wng8"
Nov 03 20:54:24.775962 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:24.775939    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-5wng8\" (UID: \"7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5wng8"
Nov 03 20:54:24.786648 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:24.786626    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t22n8\" (UniqueName: \"kubernetes.io/projected/7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1-kube-api-access-t22n8\") pod \"maas-api-revoke-sub-a4aef7d7-5wng8\" (UID: \"7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5wng8"
Nov 03 20:54:24.931945 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:24.931860    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5wng8"
Nov 03 20:54:25.212251 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:25.212124    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5wng8"]
Nov 03 20:54:25.215736 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:54:25.215709    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a1b65e5_c4b3_4d7d_9d3f_9330cc6e7eb1.slice/crio-7711399f523886356d3e0a4359ff4cd8aee715836d3a494942b2a646804827fd WatchSource:0}: Error finding container 7711399f523886356d3e0a4359ff4cd8aee715836d3a494942b2a646804827fd: Status 404 returned error can't find the container with id 7711399f523886356d3e0a4359ff4cd8aee715836d3a494942b2a646804827fd
Nov 03 20:54:26.078076 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:26.078029    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5wng8" event={"ID":"7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1","Type":"ContainerStarted","Data":"fc44b3eb4743f5461fe91be1a99d160ac1b65bfab18b42c318f1d03f38d28bde"}
Nov 03 20:54:26.078604 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:26.078106    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5wng8" event={"ID":"7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1","Type":"ContainerStarted","Data":"7711399f523886356d3e0a4359ff4cd8aee715836d3a494942b2a646804827fd"}
Nov 03 20:54:26.098545 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:26.098496    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5wng8" podStartSLOduration=2.0984824 podStartE2EDuration="2.0984824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:26.096315 +0000 UTC m=+4614.868608013" watchObservedRunningTime="2026-11-03 20:54:26.0984824 +0000 UTC m=+4614.870775414"
Nov 03 20:54:27.084931 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:27.084888    5591 generic.go:356] "Generic (PLEG): container finished" podID="7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1" containerID="fc44b3eb4743f5461fe91be1a99d160ac1b65bfab18b42c318f1d03f38d28bde" exitCode=0
Nov 03 20:54:27.085385 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:27.084937    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5wng8" event={"ID":"7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1","Type":"ContainerDied","Data":"fc44b3eb4743f5461fe91be1a99d160ac1b65bfab18b42c318f1d03f38d28bde"}
Nov 03 20:54:28.232993 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:28.232967    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5wng8"
Nov 03 20:54:28.310143 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:28.310104    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1-kube-api-access-t22n8\" (UniqueName: \"kubernetes.io/projected/7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1-kube-api-access-t22n8\") pod \"7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1\" (UID: \"7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1\") "
Nov 03 20:54:28.310384 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:28.310168    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1-maas-api-openshift-ca\") pod \"7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1\" (UID: \"7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1\") "
Nov 03 20:54:28.310384 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:28.310229    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1-maas-api-xks-ca\") pod \"7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1\" (UID: \"7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1\") "
Nov 03 20:54:28.310628 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:28.310600    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1-maas-api-openshift-ca" pod "7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1" (UID: "7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:54:28.312574 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:28.312551    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1-kube-api-access-t22n8" pod "7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1" (UID: "7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1"). InnerVolumeSpecName "kube-api-access-t22n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:54:28.312666 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:28.312578    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1-maas-api-xks-ca" pod "7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1" (UID: "7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:54:28.411489 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:28.411402    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:54:28.411489 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:28.411432    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:54:28.411489 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:28.411450    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t22n8\" (UniqueName: \"kubernetes.io/projected/7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1-kube-api-access-t22n8\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:54:28.723677 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:54:28.723580    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:54:29.096499 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:29.096463    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5wng8" event={"ID":"7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1","Type":"ContainerDied","Data":"7711399f523886356d3e0a4359ff4cd8aee715836d3a494942b2a646804827fd"}
Nov 03 20:54:29.096685 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:29.096503    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5wng8"
Nov 03 20:54:29.096685 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:29.096512    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7711399f523886356d3e0a4359ff4cd8aee715836d3a494942b2a646804827fd"
Nov 03 20:54:35.514104 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:35.514065    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:54:38.224157 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:38.224102    5591 scope.go:122] "RemoveContainer" containerID="348a2dcc9f8df8047a68bed3763387cb3e9b9b2c1a5a6cf0b7ceb71e8ba9e535"
Nov 03 20:54:40.312230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:40.308654    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d5lq5"]
Nov 03 20:54:40.312230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:40.309484    5591 cpu_manager.go:414] "containerMap: removing container" podUID="7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1" containerName="revoke-keys"
Nov 03 20:54:40.312230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:40.309506    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1" containerName="revoke-keys"
Nov 03 20:54:40.312230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:40.309612    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1" containerName="revoke-keys"
Nov 03 20:54:40.352256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:40.344435    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d5lq5"
Nov 03 20:54:40.355382 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:40.355344    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d5lq5"]
Nov 03 20:54:40.362800 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:40.362499    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:54:40.362800 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:40.362620    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:54:40.362800 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:40.362671    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:54:40.362800 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:40.362766    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 20:54:40.452985 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:40.452942    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1385f77b-5fde-4906-97ae-10b42a0dfb00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-d5lq5\" (UID: \"1385f77b-5fde-4906-97ae-10b42a0dfb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d5lq5"
Nov 03 20:54:40.453292 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:40.453265    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1385f77b-5fde-4906-97ae-10b42a0dfb00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-d5lq5\" (UID: \"1385f77b-5fde-4906-97ae-10b42a0dfb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d5lq5"
Nov 03 20:54:40.453546 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:40.453526    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b7qz\" (UniqueName: \"kubernetes.io/projected/1385f77b-5fde-4906-97ae-10b42a0dfb00-kube-api-access-9b7qz\") pod \"maas-api-revoke-sub-79805ca9-d5lq5\" (UID: \"1385f77b-5fde-4906-97ae-10b42a0dfb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d5lq5"
Nov 03 20:54:40.457235 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:40.457183    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:54:40.555232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:40.554311    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b7qz\" (UniqueName: \"kubernetes.io/projected/1385f77b-5fde-4906-97ae-10b42a0dfb00-kube-api-access-9b7qz\") pod \"maas-api-revoke-sub-79805ca9-d5lq5\" (UID: \"1385f77b-5fde-4906-97ae-10b42a0dfb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d5lq5"
Nov 03 20:54:40.555232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:40.554497    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1385f77b-5fde-4906-97ae-10b42a0dfb00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-d5lq5\" (UID: \"1385f77b-5fde-4906-97ae-10b42a0dfb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d5lq5"
Nov 03 20:54:40.555232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:40.554548    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1385f77b-5fde-4906-97ae-10b42a0dfb00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-d5lq5\" (UID: \"1385f77b-5fde-4906-97ae-10b42a0dfb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d5lq5"
Nov 03 20:54:40.556255 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:40.556226    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1385f77b-5fde-4906-97ae-10b42a0dfb00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-d5lq5\" (UID: \"1385f77b-5fde-4906-97ae-10b42a0dfb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d5lq5"
Nov 03 20:54:40.558849 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:40.558823    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1385f77b-5fde-4906-97ae-10b42a0dfb00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-d5lq5\" (UID: \"1385f77b-5fde-4906-97ae-10b42a0dfb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d5lq5"
Nov 03 20:54:40.568960 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:40.568892    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b7qz\" (UniqueName: \"kubernetes.io/projected/1385f77b-5fde-4906-97ae-10b42a0dfb00-kube-api-access-9b7qz\") pod \"maas-api-revoke-sub-79805ca9-d5lq5\" (UID: \"1385f77b-5fde-4906-97ae-10b42a0dfb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d5lq5"
Nov 03 20:54:40.670167 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:40.670130    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d5lq5"
Nov 03 20:54:41.036751 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:41.036726    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d5lq5"]
Nov 03 20:54:41.038512 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:54:41.038484    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1385f77b_5fde_4906_97ae_10b42a0dfb00.slice/crio-a372af55b71a62bf62915efa57fec23260c8a44e9e36e15e6eff2b659e6160a1 WatchSource:0}: Error finding container a372af55b71a62bf62915efa57fec23260c8a44e9e36e15e6eff2b659e6160a1: Status 404 returned error can't find the container with id a372af55b71a62bf62915efa57fec23260c8a44e9e36e15e6eff2b659e6160a1
Nov 03 20:54:41.166784 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:41.166750    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d5lq5" event={"ID":"1385f77b-5fde-4906-97ae-10b42a0dfb00","Type":"ContainerStarted","Data":"a372af55b71a62bf62915efa57fec23260c8a44e9e36e15e6eff2b659e6160a1"}
Nov 03 20:54:42.173565 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:42.173462    5591 generic.go:356] "Generic (PLEG): container finished" podID="1385f77b-5fde-4906-97ae-10b42a0dfb00" containerID="de5b14d436bb1765a99c39b6cc7580a416768269d1b342cbfc3a5176eb3f4415" exitCode=0
Nov 03 20:54:42.173565 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:42.173531    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d5lq5" event={"ID":"1385f77b-5fde-4906-97ae-10b42a0dfb00","Type":"ContainerDied","Data":"de5b14d436bb1765a99c39b6cc7580a416768269d1b342cbfc3a5176eb3f4415"}
Nov 03 20:54:43.305813 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:43.305781    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d5lq5"
Nov 03 20:54:43.382951 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:43.382922    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1385f77b-5fde-4906-97ae-10b42a0dfb00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1385f77b-5fde-4906-97ae-10b42a0dfb00-maas-api-openshift-ca\") pod \"1385f77b-5fde-4906-97ae-10b42a0dfb00\" (UID: \"1385f77b-5fde-4906-97ae-10b42a0dfb00\") "
Nov 03 20:54:43.383151 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:43.383032    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1385f77b-5fde-4906-97ae-10b42a0dfb00-kube-api-access-9b7qz\" (UniqueName: \"kubernetes.io/projected/1385f77b-5fde-4906-97ae-10b42a0dfb00-kube-api-access-9b7qz\") pod \"1385f77b-5fde-4906-97ae-10b42a0dfb00\" (UID: \"1385f77b-5fde-4906-97ae-10b42a0dfb00\") "
Nov 03 20:54:43.383151 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:43.383087    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1385f77b-5fde-4906-97ae-10b42a0dfb00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1385f77b-5fde-4906-97ae-10b42a0dfb00-maas-api-xks-ca\") pod \"1385f77b-5fde-4906-97ae-10b42a0dfb00\" (UID: \"1385f77b-5fde-4906-97ae-10b42a0dfb00\") "
Nov 03 20:54:43.383358 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:43.383331    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1385f77b-5fde-4906-97ae-10b42a0dfb00-maas-api-openshift-ca" pod "1385f77b-5fde-4906-97ae-10b42a0dfb00" (UID: "1385f77b-5fde-4906-97ae-10b42a0dfb00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:54:43.383459 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:43.383441    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1385f77b-5fde-4906-97ae-10b42a0dfb00-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:54:43.385256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:43.385227    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1385f77b-5fde-4906-97ae-10b42a0dfb00-kube-api-access-9b7qz" pod "1385f77b-5fde-4906-97ae-10b42a0dfb00" (UID: "1385f77b-5fde-4906-97ae-10b42a0dfb00"). InnerVolumeSpecName "kube-api-access-9b7qz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:54:43.385369 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:43.385250    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1385f77b-5fde-4906-97ae-10b42a0dfb00-maas-api-xks-ca" pod "1385f77b-5fde-4906-97ae-10b42a0dfb00" (UID: "1385f77b-5fde-4906-97ae-10b42a0dfb00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:54:43.485006 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:43.484918    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1385f77b-5fde-4906-97ae-10b42a0dfb00-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:54:43.485006 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:43.484953    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b7qz\" (UniqueName: \"kubernetes.io/projected/1385f77b-5fde-4906-97ae-10b42a0dfb00-kube-api-access-9b7qz\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:54:43.527761 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:43.527728    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:54:44.185126 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:44.185083    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d5lq5" event={"ID":"1385f77b-5fde-4906-97ae-10b42a0dfb00","Type":"ContainerDied","Data":"a372af55b71a62bf62915efa57fec23260c8a44e9e36e15e6eff2b659e6160a1"}
Nov 03 20:54:44.185126 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:44.185107    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d5lq5"
Nov 03 20:54:44.185126 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:44.185132    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a372af55b71a62bf62915efa57fec23260c8a44e9e36e15e6eff2b659e6160a1"
Nov 03 20:54:51.935354 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:51.935317    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:54:54.747230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:54.746169    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-x4bds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:54:54.747230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:54.746325    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-x4bds" podUID="128a2ead-106d-483e-a4e0-8e13c7c91e28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:54:54.828819 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:54.828778    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2w7p"]
Nov 03 20:54:54.829747 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:54.829720    5591 cpu_manager.go:414] "containerMap: removing container" podUID="1385f77b-5fde-4906-97ae-10b42a0dfb00" containerName="revoke-keys"
Nov 03 20:54:54.830553 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:54.830529    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1385f77b-5fde-4906-97ae-10b42a0dfb00" containerName="revoke-keys"
Nov 03 20:54:54.830845 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:54.830829    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="1385f77b-5fde-4906-97ae-10b42a0dfb00" containerName="revoke-keys"
Nov 03 20:54:54.839888 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:54.839864    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2w7p"
Nov 03 20:54:54.855229 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:54.841315    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2w7p"]
Nov 03 20:54:54.855485 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:54.846694    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 20:54:54.855610 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:54.846719    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:54:54.862601 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:54.846785    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:54:54.862601 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:54.851844    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:54:54.935230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:54.934732    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbgw8\" (UniqueName: \"kubernetes.io/projected/40236cb7-8e9d-48e0-994d-bc1580d17fd9-kube-api-access-wbgw8\") pod \"maas-api-revoke-sub-0b7e8edb-q2w7p\" (UID: \"40236cb7-8e9d-48e0-994d-bc1580d17fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2w7p"
Nov 03 20:54:54.935230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:54.934874    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40236cb7-8e9d-48e0-994d-bc1580d17fd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q2w7p\" (UID: \"40236cb7-8e9d-48e0-994d-bc1580d17fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2w7p"
Nov 03 20:54:54.935230 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:54.934991    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40236cb7-8e9d-48e0-994d-bc1580d17fd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q2w7p\" (UID: \"40236cb7-8e9d-48e0-994d-bc1580d17fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2w7p"
Nov 03 20:54:55.038310 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:55.035731    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40236cb7-8e9d-48e0-994d-bc1580d17fd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q2w7p\" (UID: \"40236cb7-8e9d-48e0-994d-bc1580d17fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2w7p"
Nov 03 20:54:55.038310 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:55.035785    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbgw8\" (UniqueName: \"kubernetes.io/projected/40236cb7-8e9d-48e0-994d-bc1580d17fd9-kube-api-access-wbgw8\") pod \"maas-api-revoke-sub-0b7e8edb-q2w7p\" (UID: \"40236cb7-8e9d-48e0-994d-bc1580d17fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2w7p"
Nov 03 20:54:55.038310 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:55.035892    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40236cb7-8e9d-48e0-994d-bc1580d17fd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q2w7p\" (UID: \"40236cb7-8e9d-48e0-994d-bc1580d17fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2w7p"
Nov 03 20:54:55.044997 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:55.043682    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40236cb7-8e9d-48e0-994d-bc1580d17fd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q2w7p\" (UID: \"40236cb7-8e9d-48e0-994d-bc1580d17fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2w7p"
Nov 03 20:54:55.044997 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:55.044541    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40236cb7-8e9d-48e0-994d-bc1580d17fd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-q2w7p\" (UID: \"40236cb7-8e9d-48e0-994d-bc1580d17fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2w7p"
Nov 03 20:54:55.056846 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:55.056814    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbgw8\" (UniqueName: \"kubernetes.io/projected/40236cb7-8e9d-48e0-994d-bc1580d17fd9-kube-api-access-wbgw8\") pod \"maas-api-revoke-sub-0b7e8edb-q2w7p\" (UID: \"40236cb7-8e9d-48e0-994d-bc1580d17fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2w7p"
Nov 03 20:54:55.170548 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:55.170507    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2w7p"
Nov 03 20:54:55.452248 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:55.452050    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2w7p"]
Nov 03 20:54:55.454668 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:54:55.454636    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40236cb7_8e9d_48e0_994d_bc1580d17fd9.slice/crio-f0dabe80b93719dd8a4afed741648a1b77446f251b5882f4c9e5db90d546dde7 WatchSource:0}: Error finding container f0dabe80b93719dd8a4afed741648a1b77446f251b5882f4c9e5db90d546dde7: Status 404 returned error can't find the container with id f0dabe80b93719dd8a4afed741648a1b77446f251b5882f4c9e5db90d546dde7
Nov 03 20:54:56.271557 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:56.271521    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2w7p" event={"ID":"40236cb7-8e9d-48e0-994d-bc1580d17fd9","Type":"ContainerStarted","Data":"f17f46a45968ef6598bafac11714400bb5ccf428af957141cd7778c571ff6e1f"}
Nov 03 20:54:56.271978 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:56.271611    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2w7p" event={"ID":"40236cb7-8e9d-48e0-994d-bc1580d17fd9","Type":"ContainerStarted","Data":"f0dabe80b93719dd8a4afed741648a1b77446f251b5882f4c9e5db90d546dde7"}
Nov 03 20:54:56.290981 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:56.290933    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2w7p" podStartSLOduration=2.2909192210000002 podStartE2EDuration="2.290919221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:56.288617882 +0000 UTC m=+4645.060910901" watchObservedRunningTime="2026-11-03 20:54:56.290919221 +0000 UTC m=+4645.063212238"
Nov 03 20:54:57.277922 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:57.277884    5591 generic.go:356] "Generic (PLEG): container finished" podID="40236cb7-8e9d-48e0-994d-bc1580d17fd9" containerID="f17f46a45968ef6598bafac11714400bb5ccf428af957141cd7778c571ff6e1f" exitCode=0
Nov 03 20:54:57.278369 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:57.277929    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2w7p" event={"ID":"40236cb7-8e9d-48e0-994d-bc1580d17fd9","Type":"ContainerDied","Data":"f17f46a45968ef6598bafac11714400bb5ccf428af957141cd7778c571ff6e1f"}
Nov 03 20:54:58.421926 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:58.421898    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2w7p"
Nov 03 20:54:58.573275 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:58.573179    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40236cb7-8e9d-48e0-994d-bc1580d17fd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40236cb7-8e9d-48e0-994d-bc1580d17fd9-maas-api-xks-ca\") pod \"40236cb7-8e9d-48e0-994d-bc1580d17fd9\" (UID: \"40236cb7-8e9d-48e0-994d-bc1580d17fd9\") "
Nov 03 20:54:58.573470 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:58.573335    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40236cb7-8e9d-48e0-994d-bc1580d17fd9-kube-api-access-wbgw8\" (UniqueName: \"kubernetes.io/projected/40236cb7-8e9d-48e0-994d-bc1580d17fd9-kube-api-access-wbgw8\") pod \"40236cb7-8e9d-48e0-994d-bc1580d17fd9\" (UID: \"40236cb7-8e9d-48e0-994d-bc1580d17fd9\") "
Nov 03 20:54:58.573470 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:58.573386    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40236cb7-8e9d-48e0-994d-bc1580d17fd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40236cb7-8e9d-48e0-994d-bc1580d17fd9-maas-api-openshift-ca\") pod \"40236cb7-8e9d-48e0-994d-bc1580d17fd9\" (UID: \"40236cb7-8e9d-48e0-994d-bc1580d17fd9\") "
Nov 03 20:54:58.573776 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:58.573744    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40236cb7-8e9d-48e0-994d-bc1580d17fd9-maas-api-openshift-ca" pod "40236cb7-8e9d-48e0-994d-bc1580d17fd9" (UID: "40236cb7-8e9d-48e0-994d-bc1580d17fd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:54:58.575456 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:58.575431    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40236cb7-8e9d-48e0-994d-bc1580d17fd9-maas-api-xks-ca" pod "40236cb7-8e9d-48e0-994d-bc1580d17fd9" (UID: "40236cb7-8e9d-48e0-994d-bc1580d17fd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:54:58.575608 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:58.575586    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40236cb7-8e9d-48e0-994d-bc1580d17fd9-kube-api-access-wbgw8" pod "40236cb7-8e9d-48e0-994d-bc1580d17fd9" (UID: "40236cb7-8e9d-48e0-994d-bc1580d17fd9"). InnerVolumeSpecName "kube-api-access-wbgw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:54:58.674554 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:58.674513    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40236cb7-8e9d-48e0-994d-bc1580d17fd9-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:54:58.674554 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:58.674546    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40236cb7-8e9d-48e0-994d-bc1580d17fd9-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:54:58.674554 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:58.674557    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbgw8\" (UniqueName: \"kubernetes.io/projected/40236cb7-8e9d-48e0-994d-bc1580d17fd9-kube-api-access-wbgw8\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:54:58.723340 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:54:58.723296    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:54:59.290169 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:59.290135    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2w7p" event={"ID":"40236cb7-8e9d-48e0-994d-bc1580d17fd9","Type":"ContainerDied","Data":"f0dabe80b93719dd8a4afed741648a1b77446f251b5882f4c9e5db90d546dde7"}
Nov 03 20:54:59.290169 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:59.290155    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2w7p"
Nov 03 20:54:59.290408 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:54:59.290184    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0dabe80b93719dd8a4afed741648a1b77446f251b5882f4c9e5db90d546dde7"
Nov 03 20:55:05.717312 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:05.717274    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:55:25.500232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:25.500178    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kxt52"]
Nov 03 20:55:25.500717 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:25.500663    5591 cpu_manager.go:414] "containerMap: removing container" podUID="40236cb7-8e9d-48e0-994d-bc1580d17fd9" containerName="revoke-keys"
Nov 03 20:55:25.500717 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:25.500681    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40236cb7-8e9d-48e0-994d-bc1580d17fd9" containerName="revoke-keys"
Nov 03 20:55:25.500826 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:25.500773    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="40236cb7-8e9d-48e0-994d-bc1580d17fd9" containerName="revoke-keys"
Nov 03 20:55:25.503766 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:25.503749    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kxt52"
Nov 03 20:55:25.509615 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:25.509073    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:55:25.510146 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:25.510114    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:55:25.510328 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:25.510189    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 20:55:25.510404 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:25.510334    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:55:25.513225 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:25.513143    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kxt52"]
Nov 03 20:55:25.633078 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:25.633046    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22e96a72-f115-4790-9990-9028c9bff291-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kxt52\" (UID: \"22e96a72-f115-4790-9990-9028c9bff291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kxt52"
Nov 03 20:55:25.633292 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:25.633104    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8kjs\" (UniqueName: \"kubernetes.io/projected/22e96a72-f115-4790-9990-9028c9bff291-kube-api-access-v8kjs\") pod \"maas-api-revoke-sub-3d4156f1-kxt52\" (UID: \"22e96a72-f115-4790-9990-9028c9bff291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kxt52"
Nov 03 20:55:25.633292 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:25.633242    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22e96a72-f115-4790-9990-9028c9bff291-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kxt52\" (UID: \"22e96a72-f115-4790-9990-9028c9bff291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kxt52"
Nov 03 20:55:25.734938 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:25.734904    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22e96a72-f115-4790-9990-9028c9bff291-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kxt52\" (UID: \"22e96a72-f115-4790-9990-9028c9bff291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kxt52"
Nov 03 20:55:25.735125 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:25.734985    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22e96a72-f115-4790-9990-9028c9bff291-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kxt52\" (UID: \"22e96a72-f115-4790-9990-9028c9bff291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kxt52"
Nov 03 20:55:25.735125 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:25.735039    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8kjs\" (UniqueName: \"kubernetes.io/projected/22e96a72-f115-4790-9990-9028c9bff291-kube-api-access-v8kjs\") pod \"maas-api-revoke-sub-3d4156f1-kxt52\" (UID: \"22e96a72-f115-4790-9990-9028c9bff291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kxt52"
Nov 03 20:55:25.735597 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:25.735575    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22e96a72-f115-4790-9990-9028c9bff291-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kxt52\" (UID: \"22e96a72-f115-4790-9990-9028c9bff291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kxt52"
Nov 03 20:55:25.737843 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:25.737818    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22e96a72-f115-4790-9990-9028c9bff291-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kxt52\" (UID: \"22e96a72-f115-4790-9990-9028c9bff291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kxt52"
Nov 03 20:55:25.747581 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:25.747556    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8kjs\" (UniqueName: \"kubernetes.io/projected/22e96a72-f115-4790-9990-9028c9bff291-kube-api-access-v8kjs\") pod \"maas-api-revoke-sub-3d4156f1-kxt52\" (UID: \"22e96a72-f115-4790-9990-9028c9bff291\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kxt52"
Nov 03 20:55:25.827225 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:25.827160    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kxt52"
Nov 03 20:55:26.104948 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:26.104924    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kxt52"]
Nov 03 20:55:26.107332 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:55:26.107304    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e96a72_f115_4790_9990_9028c9bff291.slice/crio-0f28c1e15fc9e81182d81cd58ecae5ffac55fe8f4e275431334c01ab895cb550 WatchSource:0}: Error finding container 0f28c1e15fc9e81182d81cd58ecae5ffac55fe8f4e275431334c01ab895cb550: Status 404 returned error can't find the container with id 0f28c1e15fc9e81182d81cd58ecae5ffac55fe8f4e275431334c01ab895cb550
Nov 03 20:55:26.434612 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:26.434577    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kxt52" event={"ID":"22e96a72-f115-4790-9990-9028c9bff291","Type":"ContainerStarted","Data":"764ca844cd5a949320145b8303feae4dc71e61d83a26d8ff9c9df1881e16857d"}
Nov 03 20:55:26.434814 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:26.434630    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kxt52" event={"ID":"22e96a72-f115-4790-9990-9028c9bff291","Type":"ContainerStarted","Data":"0f28c1e15fc9e81182d81cd58ecae5ffac55fe8f4e275431334c01ab895cb550"}
Nov 03 20:55:26.453721 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:26.453667    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kxt52" podStartSLOduration=1.4536545840000001 podStartE2EDuration="1.453654584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:26.451850504 +0000 UTC m=+4675.224143523" watchObservedRunningTime="2026-11-03 20:55:26.453654584 +0000 UTC m=+4675.225947623"
Nov 03 20:55:27.441617 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:27.441578    5591 generic.go:356] "Generic (PLEG): container finished" podID="22e96a72-f115-4790-9990-9028c9bff291" containerID="764ca844cd5a949320145b8303feae4dc71e61d83a26d8ff9c9df1881e16857d" exitCode=0
Nov 03 20:55:27.442032 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:27.441661    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kxt52" event={"ID":"22e96a72-f115-4790-9990-9028c9bff291","Type":"ContainerDied","Data":"764ca844cd5a949320145b8303feae4dc71e61d83a26d8ff9c9df1881e16857d"}
Nov 03 20:55:28.588666 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:28.588641    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kxt52"
Nov 03 20:55:28.663648 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:28.663621    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22e96a72-f115-4790-9990-9028c9bff291-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22e96a72-f115-4790-9990-9028c9bff291-maas-api-openshift-ca\") pod \"22e96a72-f115-4790-9990-9028c9bff291\" (UID: \"22e96a72-f115-4790-9990-9028c9bff291\") "
Nov 03 20:55:28.663648 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:28.663660    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22e96a72-f115-4790-9990-9028c9bff291-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22e96a72-f115-4790-9990-9028c9bff291-maas-api-xks-ca\") pod \"22e96a72-f115-4790-9990-9028c9bff291\" (UID: \"22e96a72-f115-4790-9990-9028c9bff291\") "
Nov 03 20:55:28.663914 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:28.663811    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22e96a72-f115-4790-9990-9028c9bff291-kube-api-access-v8kjs\" (UniqueName: \"kubernetes.io/projected/22e96a72-f115-4790-9990-9028c9bff291-kube-api-access-v8kjs\") pod \"22e96a72-f115-4790-9990-9028c9bff291\" (UID: \"22e96a72-f115-4790-9990-9028c9bff291\") "
Nov 03 20:55:28.664028 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:28.664000    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e96a72-f115-4790-9990-9028c9bff291-maas-api-openshift-ca" pod "22e96a72-f115-4790-9990-9028c9bff291" (UID: "22e96a72-f115-4790-9990-9028c9bff291"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:55:28.666042 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:28.666008    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22e96a72-f115-4790-9990-9028c9bff291-maas-api-xks-ca" pod "22e96a72-f115-4790-9990-9028c9bff291" (UID: "22e96a72-f115-4790-9990-9028c9bff291"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:55:28.666042 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:28.666024    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e96a72-f115-4790-9990-9028c9bff291-kube-api-access-v8kjs" pod "22e96a72-f115-4790-9990-9028c9bff291" (UID: "22e96a72-f115-4790-9990-9028c9bff291"). InnerVolumeSpecName "kube-api-access-v8kjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:55:28.723785 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:55:28.723696    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:55:28.764634 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:28.764602    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8kjs\" (UniqueName: \"kubernetes.io/projected/22e96a72-f115-4790-9990-9028c9bff291-kube-api-access-v8kjs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:28.764634 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:28.764631    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22e96a72-f115-4790-9990-9028c9bff291-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:28.764634 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:28.764641    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22e96a72-f115-4790-9990-9028c9bff291-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:29.453621 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:29.453581    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kxt52" event={"ID":"22e96a72-f115-4790-9990-9028c9bff291","Type":"ContainerDied","Data":"0f28c1e15fc9e81182d81cd58ecae5ffac55fe8f4e275431334c01ab895cb550"}
Nov 03 20:55:29.453621 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:29.453616    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kxt52"
Nov 03 20:55:29.453621 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:29.453630    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f28c1e15fc9e81182d81cd58ecae5ffac55fe8f4e275431334c01ab895cb550"
Nov 03 20:55:36.332011 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:36.331974    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:55:38.254429 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:38.254386    5591 scope.go:122] "RemoveContainer" containerID="37accab11ae456f62f0689fa7b91154d3ea1d768715d5bcce282dfdcb8482748"
Nov 03 20:55:38.266088 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:38.266067    5591 scope.go:122] "RemoveContainer" containerID="1c86e5bc8f8ce72d88f01b1f5660528b3a866b99e3628534dfffbe4e95642fe0"
Nov 03 20:55:40.558525 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:40.558485    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:55:41.787289 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:41.784403    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-mrwmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:55:41.787289 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:41.784494    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt" podUID="e14ca8f3-c80b-4c0b-9564-ef9734b86de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:55:42.748497 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:42.748465    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kxt52"]
Nov 03 20:55:42.751635 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:42.751602    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kxt52"]
Nov 03 20:55:43.348125 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:43.348093    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22e96a72-f115-4790-9990-9028c9bff291" path="/var/lib/kubelet/pods/22e96a72-f115-4790-9990-9028c9bff291/volumes"
Nov 03 20:55:52.804577 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:52.804542    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dw9l5"]
Nov 03 20:55:52.805067 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:52.804989    5591 cpu_manager.go:414] "containerMap: removing container" podUID="22e96a72-f115-4790-9990-9028c9bff291" containerName="revoke-keys"
Nov 03 20:55:52.805067 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:52.805001    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22e96a72-f115-4790-9990-9028c9bff291" containerName="revoke-keys"
Nov 03 20:55:52.805143 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:52.805087    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="22e96a72-f115-4790-9990-9028c9bff291" containerName="revoke-keys"
Nov 03 20:55:52.808138 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:52.808120    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dw9l5"
Nov 03 20:55:52.815263 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:52.815241    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:55:52.816052 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:52.815791    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:55:52.816052 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:52.815818    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 20:55:52.816052 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:52.815894    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:55:52.818378 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:52.818357    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dw9l5"]
Nov 03 20:55:52.927441 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:52.927399    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/116838df-d4ff-4d96-ad75-0a0d5c0c3d16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dw9l5\" (UID: \"116838df-d4ff-4d96-ad75-0a0d5c0c3d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dw9l5"
Nov 03 20:55:52.927672 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:52.927461    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxhx6\" (UniqueName: \"kubernetes.io/projected/116838df-d4ff-4d96-ad75-0a0d5c0c3d16-kube-api-access-bxhx6\") pod \"maas-api-revoke-sub-3d4156f1-dw9l5\" (UID: \"116838df-d4ff-4d96-ad75-0a0d5c0c3d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dw9l5"
Nov 03 20:55:52.927672 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:52.927598    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/116838df-d4ff-4d96-ad75-0a0d5c0c3d16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dw9l5\" (UID: \"116838df-d4ff-4d96-ad75-0a0d5c0c3d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dw9l5"
Nov 03 20:55:53.028709 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:53.028665    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/116838df-d4ff-4d96-ad75-0a0d5c0c3d16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dw9l5\" (UID: \"116838df-d4ff-4d96-ad75-0a0d5c0c3d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dw9l5"
Nov 03 20:55:53.028932 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:53.028724    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxhx6\" (UniqueName: \"kubernetes.io/projected/116838df-d4ff-4d96-ad75-0a0d5c0c3d16-kube-api-access-bxhx6\") pod \"maas-api-revoke-sub-3d4156f1-dw9l5\" (UID: \"116838df-d4ff-4d96-ad75-0a0d5c0c3d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dw9l5"
Nov 03 20:55:53.028932 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:53.028784    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/116838df-d4ff-4d96-ad75-0a0d5c0c3d16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dw9l5\" (UID: \"116838df-d4ff-4d96-ad75-0a0d5c0c3d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dw9l5"
Nov 03 20:55:53.029515 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:53.029494    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/116838df-d4ff-4d96-ad75-0a0d5c0c3d16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dw9l5\" (UID: \"116838df-d4ff-4d96-ad75-0a0d5c0c3d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dw9l5"
Nov 03 20:55:53.031169 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:53.031149    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/116838df-d4ff-4d96-ad75-0a0d5c0c3d16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dw9l5\" (UID: \"116838df-d4ff-4d96-ad75-0a0d5c0c3d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dw9l5"
Nov 03 20:55:53.041196 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:53.041173    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxhx6\" (UniqueName: \"kubernetes.io/projected/116838df-d4ff-4d96-ad75-0a0d5c0c3d16-kube-api-access-bxhx6\") pod \"maas-api-revoke-sub-3d4156f1-dw9l5\" (UID: \"116838df-d4ff-4d96-ad75-0a0d5c0c3d16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dw9l5"
Nov 03 20:55:53.123060 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:53.122974    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dw9l5"
Nov 03 20:55:53.398658 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:53.398632    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dw9l5"]
Nov 03 20:55:53.401086 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:55:53.401058    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116838df_d4ff_4d96_ad75_0a0d5c0c3d16.slice/crio-b822c285e18c2c0f8a701c42a9a29330c45dbe6ff209079ef9867ba6ccc48103 WatchSource:0}: Error finding container b822c285e18c2c0f8a701c42a9a29330c45dbe6ff209079ef9867ba6ccc48103: Status 404 returned error can't find the container with id b822c285e18c2c0f8a701c42a9a29330c45dbe6ff209079ef9867ba6ccc48103
Nov 03 20:55:53.611348 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:53.611312    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dw9l5" event={"ID":"116838df-d4ff-4d96-ad75-0a0d5c0c3d16","Type":"ContainerStarted","Data":"b822c285e18c2c0f8a701c42a9a29330c45dbe6ff209079ef9867ba6ccc48103"}
Nov 03 20:55:54.617978 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:54.617942    5591 generic.go:356] "Generic (PLEG): container finished" podID="116838df-d4ff-4d96-ad75-0a0d5c0c3d16" containerID="2ae0a58b6850ddff351468414babed9d4fefc83b3575ca520695d393f79ef97e" exitCode=0
Nov 03 20:55:54.618439 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:54.618025    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dw9l5" event={"ID":"116838df-d4ff-4d96-ad75-0a0d5c0c3d16","Type":"ContainerDied","Data":"2ae0a58b6850ddff351468414babed9d4fefc83b3575ca520695d393f79ef97e"}
Nov 03 20:55:55.762414 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:55.762387    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dw9l5"
Nov 03 20:55:55.857594 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:55.857565    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/116838df-d4ff-4d96-ad75-0a0d5c0c3d16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/116838df-d4ff-4d96-ad75-0a0d5c0c3d16-maas-api-openshift-ca\") pod \"116838df-d4ff-4d96-ad75-0a0d5c0c3d16\" (UID: \"116838df-d4ff-4d96-ad75-0a0d5c0c3d16\") "
Nov 03 20:55:55.857803 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:55.857686    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/116838df-d4ff-4d96-ad75-0a0d5c0c3d16-kube-api-access-bxhx6\" (UniqueName: \"kubernetes.io/projected/116838df-d4ff-4d96-ad75-0a0d5c0c3d16-kube-api-access-bxhx6\") pod \"116838df-d4ff-4d96-ad75-0a0d5c0c3d16\" (UID: \"116838df-d4ff-4d96-ad75-0a0d5c0c3d16\") "
Nov 03 20:55:55.857803 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:55.857768    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/116838df-d4ff-4d96-ad75-0a0d5c0c3d16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/116838df-d4ff-4d96-ad75-0a0d5c0c3d16-maas-api-xks-ca\") pod \"116838df-d4ff-4d96-ad75-0a0d5c0c3d16\" (UID: \"116838df-d4ff-4d96-ad75-0a0d5c0c3d16\") "
Nov 03 20:55:55.858006 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:55.857981    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/116838df-d4ff-4d96-ad75-0a0d5c0c3d16-maas-api-openshift-ca" pod "116838df-d4ff-4d96-ad75-0a0d5c0c3d16" (UID: "116838df-d4ff-4d96-ad75-0a0d5c0c3d16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:55:55.860145 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:55.860110    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/116838df-d4ff-4d96-ad75-0a0d5c0c3d16-maas-api-xks-ca" pod "116838df-d4ff-4d96-ad75-0a0d5c0c3d16" (UID: "116838df-d4ff-4d96-ad75-0a0d5c0c3d16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:55:55.860336 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:55.860193    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/116838df-d4ff-4d96-ad75-0a0d5c0c3d16-kube-api-access-bxhx6" pod "116838df-d4ff-4d96-ad75-0a0d5c0c3d16" (UID: "116838df-d4ff-4d96-ad75-0a0d5c0c3d16"). InnerVolumeSpecName "kube-api-access-bxhx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:55:55.958590 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:55.958501    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxhx6\" (UniqueName: \"kubernetes.io/projected/116838df-d4ff-4d96-ad75-0a0d5c0c3d16-kube-api-access-bxhx6\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:55.958590 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:55.958532    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/116838df-d4ff-4d96-ad75-0a0d5c0c3d16-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:55.958590 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:55.958542    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/116838df-d4ff-4d96-ad75-0a0d5c0c3d16-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:55:56.630556 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:56.630519    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dw9l5" event={"ID":"116838df-d4ff-4d96-ad75-0a0d5c0c3d16","Type":"ContainerDied","Data":"b822c285e18c2c0f8a701c42a9a29330c45dbe6ff209079ef9867ba6ccc48103"}
Nov 03 20:55:56.630556 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:56.630551    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dw9l5"
Nov 03 20:55:56.630815 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:55:56.630568    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b822c285e18c2c0f8a701c42a9a29330c45dbe6ff209079ef9867ba6ccc48103"
Nov 03 20:55:58.723334 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:55:58.723290    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:56:04.243050 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:04.243014    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:56:08.243301 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:08.243264    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:56:12.629614 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:12.629572    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:56:25.131173 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:25.131136    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-62bn4"]
Nov 03 20:56:25.131772 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:25.131739    5591 cpu_manager.go:414] "containerMap: removing container" podUID="116838df-d4ff-4d96-ad75-0a0d5c0c3d16" containerName="revoke-keys"
Nov 03 20:56:25.131772 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:25.131759    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="116838df-d4ff-4d96-ad75-0a0d5c0c3d16" containerName="revoke-keys"
Nov 03 20:56:25.132048 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:25.131908    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="116838df-d4ff-4d96-ad75-0a0d5c0c3d16" containerName="revoke-keys"
Nov 03 20:56:25.134997 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:25.134979    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-62bn4"
Nov 03 20:56:25.140568 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:25.140540    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:56:25.140709 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:25.140669    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:56:25.140709 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:25.140629    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:56:25.141843 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:25.141817    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 20:56:25.144984 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:25.144959    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-62bn4"]
Nov 03 20:56:25.175893 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:25.175848    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-62bn4\" (UID: \"1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-62bn4"
Nov 03 20:56:25.176085 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:25.175908    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-62bn4\" (UID: \"1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-62bn4"
Nov 03 20:56:25.176247 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:25.176102    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwjmx\" (UniqueName: \"kubernetes.io/projected/1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77-kube-api-access-dwjmx\") pod \"maas-api-revoke-sub-e2fc9c5b-62bn4\" (UID: \"1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-62bn4"
Nov 03 20:56:25.276664 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:25.276631    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-62bn4\" (UID: \"1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-62bn4"
Nov 03 20:56:25.276852 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:25.276711    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwjmx\" (UniqueName: \"kubernetes.io/projected/1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77-kube-api-access-dwjmx\") pod \"maas-api-revoke-sub-e2fc9c5b-62bn4\" (UID: \"1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-62bn4"
Nov 03 20:56:25.276852 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:25.276771    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-62bn4\" (UID: \"1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-62bn4"
Nov 03 20:56:25.277484 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:25.277413    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-62bn4\" (UID: \"1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-62bn4"
Nov 03 20:56:25.279338 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:25.279307    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-62bn4\" (UID: \"1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-62bn4"
Nov 03 20:56:25.289412 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:25.289383    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwjmx\" (UniqueName: \"kubernetes.io/projected/1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77-kube-api-access-dwjmx\") pod \"maas-api-revoke-sub-e2fc9c5b-62bn4\" (UID: \"1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-62bn4"
Nov 03 20:56:25.449935 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:25.449845    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-62bn4"
Nov 03 20:56:25.726899 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:25.726875    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-62bn4"]
Nov 03 20:56:25.836529 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:25.836487    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-62bn4" event={"ID":"1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77","Type":"ContainerStarted","Data":"04e658f22d0c2c8b885027dc2ce262e45ebbb7796d59529eead57efeddca2792"}
Nov 03 20:56:26.843030 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:26.842994    5591 generic.go:356] "Generic (PLEG): container finished" podID="1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77" containerID="a284057cf51f343dbb001f04102d88d2fa2710882f090e55a3628b8393e71665" exitCode=0
Nov 03 20:56:26.843536 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:26.843084    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-62bn4" event={"ID":"1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77","Type":"ContainerDied","Data":"a284057cf51f343dbb001f04102d88d2fa2710882f090e55a3628b8393e71665"}
Nov 03 20:56:27.995562 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:27.995532    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-62bn4"
Nov 03 20:56:28.104771 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:28.104676    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77-maas-api-openshift-ca\") pod \"1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77\" (UID: \"1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77\") "
Nov 03 20:56:28.104771 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:28.104752    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77-maas-api-xks-ca\") pod \"1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77\" (UID: \"1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77\") "
Nov 03 20:56:28.104976 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:28.104815    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77-kube-api-access-dwjmx\" (UniqueName: \"kubernetes.io/projected/1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77-kube-api-access-dwjmx\") pod \"1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77\" (UID: \"1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77\") "
Nov 03 20:56:28.105161 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:28.105123    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77-maas-api-openshift-ca" pod "1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77" (UID: "1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:56:28.107084 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:28.107056    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77-maas-api-xks-ca" pod "1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77" (UID: "1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:56:28.107180 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:28.107144    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77-kube-api-access-dwjmx" pod "1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77" (UID: "1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77"). InnerVolumeSpecName "kube-api-access-dwjmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:56:28.205863 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:28.205821    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:56:28.205863 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:28.205858    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:56:28.205863 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:28.205868    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwjmx\" (UniqueName: \"kubernetes.io/projected/1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77-kube-api-access-dwjmx\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:56:28.723752 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:56:28.723713    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:56:28.858741 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:28.858709    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-62bn4"
Nov 03 20:56:28.858956 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:28.858715    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-62bn4" event={"ID":"1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77","Type":"ContainerDied","Data":"04e658f22d0c2c8b885027dc2ce262e45ebbb7796d59529eead57efeddca2792"}
Nov 03 20:56:28.858956 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:28.858834    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04e658f22d0c2c8b885027dc2ce262e45ebbb7796d59529eead57efeddca2792"
Nov 03 20:56:35.998197 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:35.998151    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:56:45.348273 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:45.348236    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j6znv"]
Nov 03 20:56:45.350567 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:45.348677    5591 cpu_manager.go:414] "containerMap: removing container" podUID="1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77" containerName="revoke-keys"
Nov 03 20:56:45.350567 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:45.348690    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77" containerName="revoke-keys"
Nov 03 20:56:45.350567 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:45.348785    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77" containerName="revoke-keys"
Nov 03 20:56:45.351795 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:45.351778    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j6znv"
Nov 03 20:56:45.357656 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:45.357631    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:56:45.357854 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:45.357648    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:56:45.358333 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:45.358310    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:56:45.358492 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:45.358476    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 20:56:45.361162 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:45.361136    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j6znv"]
Nov 03 20:56:45.417550 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:45.417517    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d9ca25-d5a6-4fce-b0f8-11b395a188f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-j6znv\" (UID: \"48d9ca25-d5a6-4fce-b0f8-11b395a188f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j6znv"
Nov 03 20:56:45.417733 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:45.417591    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d9ca25-d5a6-4fce-b0f8-11b395a188f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-j6znv\" (UID: \"48d9ca25-d5a6-4fce-b0f8-11b395a188f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j6znv"
Nov 03 20:56:45.417733 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:45.417636    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfx98\" (UniqueName: \"kubernetes.io/projected/48d9ca25-d5a6-4fce-b0f8-11b395a188f5-kube-api-access-bfx98\") pod \"maas-api-revoke-sub-5a29efb3-j6znv\" (UID: \"48d9ca25-d5a6-4fce-b0f8-11b395a188f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j6znv"
Nov 03 20:56:45.518608 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:45.518568    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d9ca25-d5a6-4fce-b0f8-11b395a188f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-j6znv\" (UID: \"48d9ca25-d5a6-4fce-b0f8-11b395a188f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j6znv"
Nov 03 20:56:45.518608 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:45.518609    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d9ca25-d5a6-4fce-b0f8-11b395a188f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-j6znv\" (UID: \"48d9ca25-d5a6-4fce-b0f8-11b395a188f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j6znv"
Nov 03 20:56:45.518844 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:45.518647    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfx98\" (UniqueName: \"kubernetes.io/projected/48d9ca25-d5a6-4fce-b0f8-11b395a188f5-kube-api-access-bfx98\") pod \"maas-api-revoke-sub-5a29efb3-j6znv\" (UID: \"48d9ca25-d5a6-4fce-b0f8-11b395a188f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j6znv"
Nov 03 20:56:45.519439 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:45.519419    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d9ca25-d5a6-4fce-b0f8-11b395a188f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-j6znv\" (UID: \"48d9ca25-d5a6-4fce-b0f8-11b395a188f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j6znv"
Nov 03 20:56:45.521163 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:45.521141    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d9ca25-d5a6-4fce-b0f8-11b395a188f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-j6znv\" (UID: \"48d9ca25-d5a6-4fce-b0f8-11b395a188f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j6znv"
Nov 03 20:56:45.531572 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:45.531546    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfx98\" (UniqueName: \"kubernetes.io/projected/48d9ca25-d5a6-4fce-b0f8-11b395a188f5-kube-api-access-bfx98\") pod \"maas-api-revoke-sub-5a29efb3-j6znv\" (UID: \"48d9ca25-d5a6-4fce-b0f8-11b395a188f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j6znv"
Nov 03 20:56:45.674025 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:45.673935    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j6znv"
Nov 03 20:56:45.950642 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:45.950612    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j6znv"]
Nov 03 20:56:45.952783 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:56:45.952738    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d9ca25_d5a6_4fce_b0f8_11b395a188f5.slice/crio-32c7c8274a47abc72146617c74862fe221f39170db41c19ccb48669a8f617978 WatchSource:0}: Error finding container 32c7c8274a47abc72146617c74862fe221f39170db41c19ccb48669a8f617978: Status 404 returned error can't find the container with id 32c7c8274a47abc72146617c74862fe221f39170db41c19ccb48669a8f617978
Nov 03 20:56:45.965996 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:45.965959    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j6znv" event={"ID":"48d9ca25-d5a6-4fce-b0f8-11b395a188f5","Type":"ContainerStarted","Data":"32c7c8274a47abc72146617c74862fe221f39170db41c19ccb48669a8f617978"}
Nov 03 20:56:46.973256 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:46.973198    5591 generic.go:356] "Generic (PLEG): container finished" podID="48d9ca25-d5a6-4fce-b0f8-11b395a188f5" containerID="f32909444d05585d8ee8b2a65bbb165312965d9c0655d0cbdf53435f23a3cba9" exitCode=0
Nov 03 20:56:46.973677 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:46.973254    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j6znv" event={"ID":"48d9ca25-d5a6-4fce-b0f8-11b395a188f5","Type":"ContainerDied","Data":"f32909444d05585d8ee8b2a65bbb165312965d9c0655d0cbdf53435f23a3cba9"}
Nov 03 20:56:48.116472 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:48.116447    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j6znv"
Nov 03 20:56:48.140395 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:48.140363    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48d9ca25-d5a6-4fce-b0f8-11b395a188f5-kube-api-access-bfx98\" (UniqueName: \"kubernetes.io/projected/48d9ca25-d5a6-4fce-b0f8-11b395a188f5-kube-api-access-bfx98\") pod \"48d9ca25-d5a6-4fce-b0f8-11b395a188f5\" (UID: \"48d9ca25-d5a6-4fce-b0f8-11b395a188f5\") "
Nov 03 20:56:48.140589 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:48.140455    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48d9ca25-d5a6-4fce-b0f8-11b395a188f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d9ca25-d5a6-4fce-b0f8-11b395a188f5-maas-api-xks-ca\") pod \"48d9ca25-d5a6-4fce-b0f8-11b395a188f5\" (UID: \"48d9ca25-d5a6-4fce-b0f8-11b395a188f5\") "
Nov 03 20:56:48.140589 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:48.140511    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48d9ca25-d5a6-4fce-b0f8-11b395a188f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d9ca25-d5a6-4fce-b0f8-11b395a188f5-maas-api-openshift-ca\") pod \"48d9ca25-d5a6-4fce-b0f8-11b395a188f5\" (UID: \"48d9ca25-d5a6-4fce-b0f8-11b395a188f5\") "
Nov 03 20:56:48.140993 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:48.140962    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48d9ca25-d5a6-4fce-b0f8-11b395a188f5-maas-api-openshift-ca" pod "48d9ca25-d5a6-4fce-b0f8-11b395a188f5" (UID: "48d9ca25-d5a6-4fce-b0f8-11b395a188f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:56:48.142690 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:48.142659    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48d9ca25-d5a6-4fce-b0f8-11b395a188f5-kube-api-access-bfx98" pod "48d9ca25-d5a6-4fce-b0f8-11b395a188f5" (UID: "48d9ca25-d5a6-4fce-b0f8-11b395a188f5"). InnerVolumeSpecName "kube-api-access-bfx98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:56:48.142935 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:48.142919    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48d9ca25-d5a6-4fce-b0f8-11b395a188f5-maas-api-xks-ca" pod "48d9ca25-d5a6-4fce-b0f8-11b395a188f5" (UID: "48d9ca25-d5a6-4fce-b0f8-11b395a188f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:56:48.241432 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:48.241350    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfx98\" (UniqueName: \"kubernetes.io/projected/48d9ca25-d5a6-4fce-b0f8-11b395a188f5-kube-api-access-bfx98\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:56:48.241432 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:48.241381    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48d9ca25-d5a6-4fce-b0f8-11b395a188f5-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:56:48.241432 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:48.241392    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48d9ca25-d5a6-4fce-b0f8-11b395a188f5-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:56:48.986151 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:48.986122    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j6znv"
Nov 03 20:56:48.986355 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:48.986115    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j6znv" event={"ID":"48d9ca25-d5a6-4fce-b0f8-11b395a188f5","Type":"ContainerDied","Data":"32c7c8274a47abc72146617c74862fe221f39170db41c19ccb48669a8f617978"}
Nov 03 20:56:48.986355 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:48.986273    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32c7c8274a47abc72146617c74862fe221f39170db41c19ccb48669a8f617978"
Nov 03 20:56:56.212448 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:56:56.212411    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:56:58.723509 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:56:58.723468    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:57:01.046758 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:01.046723    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dw9l5"]
Nov 03 20:57:01.049672 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:01.049641    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dw9l5"]
Nov 03 20:57:01.075274 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:01.075246    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8njpn"]
Nov 03 20:57:01.075681 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:01.075667    5591 cpu_manager.go:414] "containerMap: removing container" podUID="48d9ca25-d5a6-4fce-b0f8-11b395a188f5" containerName="revoke-keys"
Nov 03 20:57:01.075778 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:01.075682    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48d9ca25-d5a6-4fce-b0f8-11b395a188f5" containerName="revoke-keys"
Nov 03 20:57:01.075778 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:01.075761    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="48d9ca25-d5a6-4fce-b0f8-11b395a188f5" containerName="revoke-keys"
Nov 03 20:57:01.079325 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:01.079299    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8njpn"
Nov 03 20:57:01.086767 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:01.086747    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:57:01.086898 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:01.086825    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:57:01.086975 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:01.086951    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 20:57:01.087042 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:01.087002    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:57:01.089734 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:01.089707    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8njpn"]
Nov 03 20:57:01.177811 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:01.177774    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40200261-61e0-4f57-9d8b-2b16da0ede93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8njpn\" (UID: \"40200261-61e0-4f57-9d8b-2b16da0ede93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8njpn"
Nov 03 20:57:01.177811 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:01.177814    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40200261-61e0-4f57-9d8b-2b16da0ede93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8njpn\" (UID: \"40200261-61e0-4f57-9d8b-2b16da0ede93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8njpn"
Nov 03 20:57:01.178047 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:01.177961    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrm7v\" (UniqueName: \"kubernetes.io/projected/40200261-61e0-4f57-9d8b-2b16da0ede93-kube-api-access-xrm7v\") pod \"maas-api-revoke-sub-3d4156f1-8njpn\" (UID: \"40200261-61e0-4f57-9d8b-2b16da0ede93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8njpn"
Nov 03 20:57:01.278995 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:01.278958    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40200261-61e0-4f57-9d8b-2b16da0ede93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8njpn\" (UID: \"40200261-61e0-4f57-9d8b-2b16da0ede93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8njpn"
Nov 03 20:57:01.279195 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:01.279001    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40200261-61e0-4f57-9d8b-2b16da0ede93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8njpn\" (UID: \"40200261-61e0-4f57-9d8b-2b16da0ede93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8njpn"
Nov 03 20:57:01.279195 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:01.279117    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrm7v\" (UniqueName: \"kubernetes.io/projected/40200261-61e0-4f57-9d8b-2b16da0ede93-kube-api-access-xrm7v\") pod \"maas-api-revoke-sub-3d4156f1-8njpn\" (UID: \"40200261-61e0-4f57-9d8b-2b16da0ede93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8njpn"
Nov 03 20:57:01.279724 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:01.279703    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40200261-61e0-4f57-9d8b-2b16da0ede93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8njpn\" (UID: \"40200261-61e0-4f57-9d8b-2b16da0ede93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8njpn"
Nov 03 20:57:01.281494 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:01.281475    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40200261-61e0-4f57-9d8b-2b16da0ede93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8njpn\" (UID: \"40200261-61e0-4f57-9d8b-2b16da0ede93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8njpn"
Nov 03 20:57:01.291768 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:01.291743    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrm7v\" (UniqueName: \"kubernetes.io/projected/40200261-61e0-4f57-9d8b-2b16da0ede93-kube-api-access-xrm7v\") pod \"maas-api-revoke-sub-3d4156f1-8njpn\" (UID: \"40200261-61e0-4f57-9d8b-2b16da0ede93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8njpn"
Nov 03 20:57:01.348379 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:01.348302    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="116838df-d4ff-4d96-ad75-0a0d5c0c3d16" path="/var/lib/kubelet/pods/116838df-d4ff-4d96-ad75-0a0d5c0c3d16/volumes"
Nov 03 20:57:01.393741 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:01.393711    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8njpn"
Nov 03 20:57:01.683539 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:01.683505    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8njpn"]
Nov 03 20:57:01.685782 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:57:01.685754    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40200261_61e0_4f57_9d8b_2b16da0ede93.slice/crio-3f823bb1889be26994f4376429fd6b04914be09eac5ae6eb88c3c6d7ee249fed WatchSource:0}: Error finding container 3f823bb1889be26994f4376429fd6b04914be09eac5ae6eb88c3c6d7ee249fed: Status 404 returned error can't find the container with id 3f823bb1889be26994f4376429fd6b04914be09eac5ae6eb88c3c6d7ee249fed
Nov 03 20:57:02.072480 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:02.072439    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8njpn" event={"ID":"40200261-61e0-4f57-9d8b-2b16da0ede93","Type":"ContainerStarted","Data":"06b9d0e200b23e89e08757d7e56fe8ca662934aa772bf28e600b7768a3ed54fe"}
Nov 03 20:57:02.072905 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:02.072499    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8njpn" event={"ID":"40200261-61e0-4f57-9d8b-2b16da0ede93","Type":"ContainerStarted","Data":"3f823bb1889be26994f4376429fd6b04914be09eac5ae6eb88c3c6d7ee249fed"}
Nov 03 20:57:02.092062 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:02.092010    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8njpn" podStartSLOduration=1.091998004 podStartE2EDuration="1.091998004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:02.090829189 +0000 UTC m=+4770.863122202" watchObservedRunningTime="2026-11-03 20:57:02.091998004 +0000 UTC m=+4770.864291021"
Nov 03 20:57:03.079218 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:03.079169    5591 generic.go:356] "Generic (PLEG): container finished" podID="40200261-61e0-4f57-9d8b-2b16da0ede93" containerID="06b9d0e200b23e89e08757d7e56fe8ca662934aa772bf28e600b7768a3ed54fe" exitCode=0
Nov 03 20:57:03.079624 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:03.079263    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8njpn" event={"ID":"40200261-61e0-4f57-9d8b-2b16da0ede93","Type":"ContainerDied","Data":"06b9d0e200b23e89e08757d7e56fe8ca662934aa772bf28e600b7768a3ed54fe"}
Nov 03 20:57:04.224875 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:04.224851    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8njpn"
Nov 03 20:57:04.305058 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:04.305026    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40200261-61e0-4f57-9d8b-2b16da0ede93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40200261-61e0-4f57-9d8b-2b16da0ede93-maas-api-openshift-ca\") pod \"40200261-61e0-4f57-9d8b-2b16da0ede93\" (UID: \"40200261-61e0-4f57-9d8b-2b16da0ede93\") "
Nov 03 20:57:04.305297 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:04.305110    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40200261-61e0-4f57-9d8b-2b16da0ede93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40200261-61e0-4f57-9d8b-2b16da0ede93-maas-api-xks-ca\") pod \"40200261-61e0-4f57-9d8b-2b16da0ede93\" (UID: \"40200261-61e0-4f57-9d8b-2b16da0ede93\") "
Nov 03 20:57:04.305297 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:04.305146    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40200261-61e0-4f57-9d8b-2b16da0ede93-kube-api-access-xrm7v\" (UniqueName: \"kubernetes.io/projected/40200261-61e0-4f57-9d8b-2b16da0ede93-kube-api-access-xrm7v\") pod \"40200261-61e0-4f57-9d8b-2b16da0ede93\" (UID: \"40200261-61e0-4f57-9d8b-2b16da0ede93\") "
Nov 03 20:57:04.305463 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:04.305413    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40200261-61e0-4f57-9d8b-2b16da0ede93-maas-api-openshift-ca" pod "40200261-61e0-4f57-9d8b-2b16da0ede93" (UID: "40200261-61e0-4f57-9d8b-2b16da0ede93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:57:04.307480 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:04.307455    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40200261-61e0-4f57-9d8b-2b16da0ede93-maas-api-xks-ca" pod "40200261-61e0-4f57-9d8b-2b16da0ede93" (UID: "40200261-61e0-4f57-9d8b-2b16da0ede93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:57:04.307583 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:04.307487    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40200261-61e0-4f57-9d8b-2b16da0ede93-kube-api-access-xrm7v" pod "40200261-61e0-4f57-9d8b-2b16da0ede93" (UID: "40200261-61e0-4f57-9d8b-2b16da0ede93"). InnerVolumeSpecName "kube-api-access-xrm7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:57:04.406049 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:04.405969    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40200261-61e0-4f57-9d8b-2b16da0ede93-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:04.406049 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:04.405999    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40200261-61e0-4f57-9d8b-2b16da0ede93-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:04.406049 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:04.406009    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrm7v\" (UniqueName: \"kubernetes.io/projected/40200261-61e0-4f57-9d8b-2b16da0ede93-kube-api-access-xrm7v\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:57:05.092348 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:05.092309    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8njpn" event={"ID":"40200261-61e0-4f57-9d8b-2b16da0ede93","Type":"ContainerDied","Data":"3f823bb1889be26994f4376429fd6b04914be09eac5ae6eb88c3c6d7ee249fed"}
Nov 03 20:57:05.092576 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:05.092370    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f823bb1889be26994f4376429fd6b04914be09eac5ae6eb88c3c6d7ee249fed"
Nov 03 20:57:05.092576 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:05.092319    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8njpn"
Nov 03 20:57:11.024225 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:11.024174    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-2cnnn"]
Nov 03 20:57:11.027403 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:11.027378    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-2cnnn"]
Nov 03 20:57:11.347289 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:11.347173    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2aeeb83-68ec-44c3-84e7-91350fb5d744" path="/var/lib/kubelet/pods/c2aeeb83-68ec-44c3-84e7-91350fb5d744/volumes"
Nov 03 20:57:12.519543 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:12.519503    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:57:16.424316 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.424247    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"]
Nov 03 20:57:16.425328 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.425308    5591 cpu_manager.go:414] "containerMap: removing container" podUID="40200261-61e0-4f57-9d8b-2b16da0ede93" containerName="revoke-keys"
Nov 03 20:57:16.425397 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.425330    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40200261-61e0-4f57-9d8b-2b16da0ede93" containerName="revoke-keys"
Nov 03 20:57:16.425438 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.425404    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="40200261-61e0-4f57-9d8b-2b16da0ede93" containerName="revoke-keys"
Nov 03 20:57:16.430228 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.429776    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.435512 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.435484    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-4ca23e-openshift-default-dockercfg-c2j22\""
Nov 03 20:57:16.442704 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.442675    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"]
Nov 03 20:57:16.564864 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.564829    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-credential-socket\") pod \"e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") " pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.565077 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.564884    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-workload-certs\") pod \"e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") " pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.565077 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.564907    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-envoy\") pod \"e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") " pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.565077 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.565024    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-workload-socket\") pod \"e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") " pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.565328 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.565096    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-data\") pod \"e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") " pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.565328 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.565150    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtdhq\" (UniqueName: \"kubernetes.io/projected/f10a9b94-0e91-422a-a65d-4e58c0452d6e-kube-api-access-xtdhq\") pod \"e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") " pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.565328 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.565197    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istiod-ca-cert\") pod \"e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") " pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.565473 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.565345    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-token\") pod \"e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") " pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.565473 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.565376    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-podinfo\") pod \"e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") " pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.666504 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.666463    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-token\") pod \"e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") " pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.666504 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.666510    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-podinfo\") pod \"e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") " pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.666869 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.666541    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-credential-socket\") pod \"e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") " pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.666869 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.666575    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-workload-certs\") pod \"e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") " pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.666869 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.666597    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-envoy\") pod \"e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") " pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.666869 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.666621    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-workload-socket\") pod \"e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") " pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.666869 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.666637    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-data\") pod \"e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") " pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.666869 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.666670    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtdhq\" (UniqueName: \"kubernetes.io/projected/f10a9b94-0e91-422a-a65d-4e58c0452d6e-kube-api-access-xtdhq\") pod \"e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") " pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.666869 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.666706    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istiod-ca-cert\") pod \"e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") " pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.667321 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.667045    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-credential-socket\") pod \"e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") " pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.667321 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.667177    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-workload-certs\") pod \"e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") " pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.667321 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.667311    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-data\") pod \"e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") " pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.667468 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.667359    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-workload-socket\") pod \"e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") " pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.667674 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.667655    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istiod-ca-cert\") pod \"e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") " pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.669159 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.669139    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-envoy\") pod \"e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") " pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.669428 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.669411    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-podinfo\") pod \"e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") " pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.679540 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.679471    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-token\") pod \"e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") " pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.679726 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.679700    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtdhq\" (UniqueName: \"kubernetes.io/projected/f10a9b94-0e91-422a-a65d-4e58c0452d6e-kube-api-access-xtdhq\") pod \"e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") " pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:16.746975 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:16.746933    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:17.033357 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:17.033328    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"]
Nov 03 20:57:17.035466 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:57:17.035435    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10a9b94_0e91_422a_a65d_4e58c0452d6e.slice/crio-82c37cde8f796407346336000e92070578cb20dfb4dfa433f83ecdbd7689e0cb WatchSource:0}: Error finding container 82c37cde8f796407346336000e92070578cb20dfb4dfa433f83ecdbd7689e0cb: Status 404 returned error can't find the container with id 82c37cde8f796407346336000e92070578cb20dfb4dfa433f83ecdbd7689e0cb
Nov 03 20:57:17.037632 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:17.037595    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:57:17.037751 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:17.037702    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:57:17.037817 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:17.037746    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:57:17.179973 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:17.179927    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw" event={"ID":"f10a9b94-0e91-422a-a65d-4e58c0452d6e","Type":"ContainerStarted","Data":"da5f16364b96cf875494b75424d73b7d543ccbdedce10de148a1fca4e2f0f037"}
Nov 03 20:57:17.180146 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:17.180001    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw" event={"ID":"f10a9b94-0e91-422a-a65d-4e58c0452d6e","Type":"ContainerStarted","Data":"82c37cde8f796407346336000e92070578cb20dfb4dfa433f83ecdbd7689e0cb"}
Nov 03 20:57:17.204219 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:17.204154    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw" podStartSLOduration=1.2041352029999999 podStartE2EDuration="1.204135203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:17.202098593 +0000 UTC m=+4785.974391611" watchObservedRunningTime="2026-11-03 20:57:17.204135203 +0000 UTC m=+4785.976428221"
Nov 03 20:57:17.553502 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:17.553461    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:57:17.747917 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:17.747878    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:18.752995 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:18.752962    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:19.190652 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:19.190622    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:19.191577 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:19.191556    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:20.793914 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:20.793877    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x"]
Nov 03 20:57:20.797805 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:20.797785    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x"
Nov 03 20:57:20.805723 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:20.805688    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-4ca23e\""
Nov 03 20:57:20.806893 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:20.806871    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-963f71df-dockercfg-lljhl\""
Nov 03 20:57:20.809303 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:20.809275    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k"]
Nov 03 20:57:20.813407 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:20.813385    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x"]
Nov 03 20:57:20.813541 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:20.813481    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k"
Nov 03 20:57:20.824583 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:20.824552    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k"]
Nov 03 20:57:20.908231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:20.908166    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2fcf\" (UniqueName: \"kubernetes.io/projected/0f21c382-40db-4224-b589-a52a37298a4e-kube-api-access-k2fcf\") pod \"payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x\" (UID: \"0f21c382-40db-4224-b589-a52a37298a4e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x"
Nov 03 20:57:20.908414 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:20.908267    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f21c382-40db-4224-b589-a52a37298a4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x\" (UID: \"0f21c382-40db-4224-b589-a52a37298a4e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x"
Nov 03 20:57:20.908414 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:20.908328    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749a27eb-0c3d-4789-908a-180c7bb01e58-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k\" (UID: \"749a27eb-0c3d-4789-908a-180c7bb01e58\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k"
Nov 03 20:57:20.908414 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:20.908367    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvhqf\" (UniqueName: \"kubernetes.io/projected/749a27eb-0c3d-4789-908a-180c7bb01e58-kube-api-access-nvhqf\") pod \"payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k\" (UID: \"749a27eb-0c3d-4789-908a-180c7bb01e58\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k"
Nov 03 20:57:21.009519 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:21.009478    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2fcf\" (UniqueName: \"kubernetes.io/projected/0f21c382-40db-4224-b589-a52a37298a4e-kube-api-access-k2fcf\") pod \"payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x\" (UID: \"0f21c382-40db-4224-b589-a52a37298a4e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x"
Nov 03 20:57:21.009732 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:21.009533    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f21c382-40db-4224-b589-a52a37298a4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x\" (UID: \"0f21c382-40db-4224-b589-a52a37298a4e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x"
Nov 03 20:57:21.009732 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:21.009558    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749a27eb-0c3d-4789-908a-180c7bb01e58-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k\" (UID: \"749a27eb-0c3d-4789-908a-180c7bb01e58\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k"
Nov 03 20:57:21.009732 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:21.009580    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvhqf\" (UniqueName: \"kubernetes.io/projected/749a27eb-0c3d-4789-908a-180c7bb01e58-kube-api-access-nvhqf\") pod \"payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k\" (UID: \"749a27eb-0c3d-4789-908a-180c7bb01e58\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k"
Nov 03 20:57:21.010430 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:21.010402    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f21c382-40db-4224-b589-a52a37298a4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x\" (UID: \"0f21c382-40db-4224-b589-a52a37298a4e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x"
Nov 03 20:57:21.010430 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:21.010422    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749a27eb-0c3d-4789-908a-180c7bb01e58-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k\" (UID: \"749a27eb-0c3d-4789-908a-180c7bb01e58\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k"
Nov 03 20:57:21.023704 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:21.023674    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvhqf\" (UniqueName: \"kubernetes.io/projected/749a27eb-0c3d-4789-908a-180c7bb01e58-kube-api-access-nvhqf\") pod \"payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k\" (UID: \"749a27eb-0c3d-4789-908a-180c7bb01e58\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k"
Nov 03 20:57:21.028558 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:21.028531    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2fcf\" (UniqueName: \"kubernetes.io/projected/0f21c382-40db-4224-b589-a52a37298a4e-kube-api-access-k2fcf\") pod \"payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x\" (UID: \"0f21c382-40db-4224-b589-a52a37298a4e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x"
Nov 03 20:57:21.112475 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:21.112377    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x"
Nov 03 20:57:21.130875 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:21.130841    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k"
Nov 03 20:57:21.409848 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:21.400901    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x"]
Nov 03 20:57:21.415095 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:57:21.415061    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f21c382_40db_4224_b589_a52a37298a4e.slice/crio-53041918be98b97c7d16113ef24ec143871e3914fc5bac2bfed7a68c927eef41 WatchSource:0}: Error finding container 53041918be98b97c7d16113ef24ec143871e3914fc5bac2bfed7a68c927eef41: Status 404 returned error can't find the container with id 53041918be98b97c7d16113ef24ec143871e3914fc5bac2bfed7a68c927eef41
Nov 03 20:57:21.431877 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:21.431851    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k"]
Nov 03 20:57:21.433719 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:57:21.433690    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod749a27eb_0c3d_4789_908a_180c7bb01e58.slice/crio-f4f4a339e275ee5d7763a9f20bcf612841a54d499c615e17a7e33b930be9d835 WatchSource:0}: Error finding container f4f4a339e275ee5d7763a9f20bcf612841a54d499c615e17a7e33b930be9d835: Status 404 returned error can't find the container with id f4f4a339e275ee5d7763a9f20bcf612841a54d499c615e17a7e33b930be9d835
Nov 03 20:57:22.211525 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:22.211480    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x" event={"ID":"0f21c382-40db-4224-b589-a52a37298a4e","Type":"ContainerStarted","Data":"b4542a41d40b2a969ebb6cec8bb932954d39795d90286b7ab177b7ab0f329109"}
Nov 03 20:57:22.212057 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:22.211550    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x" event={"ID":"0f21c382-40db-4224-b589-a52a37298a4e","Type":"ContainerStarted","Data":"53041918be98b97c7d16113ef24ec143871e3914fc5bac2bfed7a68c927eef41"}
Nov 03 20:57:22.212057 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:22.211691    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x"
Nov 03 20:57:22.213157 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:22.213132    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k" event={"ID":"749a27eb-0c3d-4789-908a-180c7bb01e58","Type":"ContainerStarted","Data":"de5ed29a6fb2f912e39ab4a612d733b117c5b8aa8815023e587a17de7eb79082"}
Nov 03 20:57:22.213279 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:22.213179    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k" event={"ID":"749a27eb-0c3d-4789-908a-180c7bb01e58","Type":"ContainerStarted","Data":"f4f4a339e275ee5d7763a9f20bcf612841a54d499c615e17a7e33b930be9d835"}
Nov 03 20:57:22.213279 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:22.213270    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k"
Nov 03 20:57:22.231659 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:22.231593    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x" podStartSLOduration=2.231579542 podStartE2EDuration="2.231579542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:22.228523488 +0000 UTC m=+4791.000816506" watchObservedRunningTime="2026-11-03 20:57:22.231579542 +0000 UTC m=+4791.003872560"
Nov 03 20:57:22.247414 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:22.247364    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k" podStartSLOduration=2.247349741 podStartE2EDuration="2.247349741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:22.245865047 +0000 UTC m=+4791.018158066" watchObservedRunningTime="2026-11-03 20:57:22.247349741 +0000 UTC m=+4791.019642766"
Nov 03 20:57:23.410587 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:23.410548    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:57:28.732835 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:57:28.732797    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:57:32.020680 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:32.020645    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlcf9"]
Nov 03 20:57:32.025497 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:32.025468    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-nlcf9"]
Nov 03 20:57:33.221737 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:33.221658    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k"
Nov 03 20:57:33.221737 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:33.221723    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x"
Nov 03 20:57:33.279433 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:33.279397    5591 generic.go:356] "Generic (PLEG): container finished" podID="f10a9b94-0e91-422a-a65d-4e58c0452d6e" containerID="da5f16364b96cf875494b75424d73b7d543ccbdedce10de148a1fca4e2f0f037" exitCode=0
Nov 03 20:57:33.279643 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:33.279502    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw" event={"ID":"f10a9b94-0e91-422a-a65d-4e58c0452d6e","Type":"ContainerDied","Data":"da5f16364b96cf875494b75424d73b7d543ccbdedce10de148a1fca4e2f0f037"}
Nov 03 20:57:33.280049 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:33.280024    5591 scope.go:122] "RemoveContainer" containerID="da5f16364b96cf875494b75424d73b7d543ccbdedce10de148a1fca4e2f0f037"
Nov 03 20:57:33.281538 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:33.281506    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:57:33.281671 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:33.281594    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:57:33.281671 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:33.281643    5591 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 03 20:57:33.348618 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:33.348589    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65cdbcb2-4ccd-4c5c-a1de-8ee225bea762" path="/var/lib/kubelet/pods/65cdbcb2-4ccd-4c5c-a1de-8ee225bea762/volumes"
Nov 03 20:57:34.196231 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:34.194317    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:34.263364 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:34.263339    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 20:57:34.273859 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:34.273837    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 20:57:34.308452 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:34.308418    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw" event={"ID":"f10a9b94-0e91-422a-a65d-4e58c0452d6e","Type":"ContainerStarted","Data":"38946927e31d006fc8e5dd45cd259e03b11f6e2457a13dd57ea9f8379fe3a551"}
Nov 03 20:57:34.747170 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:34.747139    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:34.748317 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:34.748283    5591 patch_prober.go:29] interesting pod/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body=
Nov 03 20:57:34.748491 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:34.748356    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw" podUID="f10a9b94-0e91-422a-a65d-4e58c0452d6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused"
Nov 03 20:57:35.747865 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:35.747823    5591 patch_prober.go:29] interesting pod/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body=
Nov 03 20:57:35.748327 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:35.747895    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw" podUID="f10a9b94-0e91-422a-a65d-4e58c0452d6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused"
Nov 03 20:57:36.747365 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:36.747332    5591 patch_prober.go:29] interesting pod/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body=
Nov 03 20:57:36.747547 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:36.747395    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw" podUID="f10a9b94-0e91-422a-a65d-4e58c0452d6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused"
Nov 03 20:57:37.748351 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:37.748314    5591 patch_prober.go:29] interesting pod/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body=
Nov 03 20:57:37.748803 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:37.748375    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw" podUID="f10a9b94-0e91-422a-a65d-4e58c0452d6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused"
Nov 03 20:57:38.104303 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:38.104267    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:57:38.747732 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:38.747692    5591 patch_prober.go:29] interesting pod/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused" start-of-body=
Nov 03 20:57:38.747930 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:38.747762    5591 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw" podUID="f10a9b94-0e91-422a-a65d-4e58c0452d6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.57:15021/healthz/ready\": dial tcp 10.134.1.57:15021: connect: connection refused"
Nov 03 20:57:39.815794 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:39.815756    5591 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:39.816345 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:39.816322    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:39.817381 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:39.817357    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:57:54.005745 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:54.005707    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:57:58.723511 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:57:58.723466    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:57:59.023424 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:59.023379    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gk78w"]
Nov 03 20:57:59.025490 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:59.025466    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-gk78w"]
Nov 03 20:57:59.347577 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:57:59.347492    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31d5bf72-336a-4985-8906-5e2b8ae68345" path="/var/lib/kubelet/pods/31d5bf72-336a-4985-8906-5e2b8ae68345/volumes"
Nov 03 20:58:10.027340 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:10.027299    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w6hg6"]
Nov 03 20:58:10.029445 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:10.029414    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w6hg6"]
Nov 03 20:58:11.348721 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:11.348685    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="916f881b-1b9b-4be6-a507-8ac6313baf29" path="/var/lib/kubelet/pods/916f881b-1b9b-4be6-a507-8ac6313baf29/volumes"
Nov 03 20:58:12.105662 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:12.105622    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1fba839-cxn8c"]
Nov 03 20:58:12.111396 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:12.111372    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fba839-cxn8c"
Nov 03 20:58:12.117485 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:12.116927    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:58:12.117485 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:12.116958    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:58:12.117485 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:12.116991    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:58:12.117485 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:12.116932    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 20:58:12.119240 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:12.119195    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1fba839-cxn8c"]
Nov 03 20:58:12.173394 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:12.173342    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1fba839-cxn8c\" (UID: \"7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fba839-cxn8c"
Nov 03 20:58:12.173624 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:12.173526    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1fba839-cxn8c\" (UID: \"7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fba839-cxn8c"
Nov 03 20:58:12.173698 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:12.173631    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc6pp\" (UniqueName: \"kubernetes.io/projected/7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6-kube-api-access-cc6pp\") pod \"maas-api-revoke-sub-f1fba839-cxn8c\" (UID: \"7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fba839-cxn8c"
Nov 03 20:58:12.274740 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:12.274677    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1fba839-cxn8c\" (UID: \"7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fba839-cxn8c"
Nov 03 20:58:12.274957 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:12.274772    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc6pp\" (UniqueName: \"kubernetes.io/projected/7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6-kube-api-access-cc6pp\") pod \"maas-api-revoke-sub-f1fba839-cxn8c\" (UID: \"7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fba839-cxn8c"
Nov 03 20:58:12.274957 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:12.274826    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1fba839-cxn8c\" (UID: \"7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fba839-cxn8c"
Nov 03 20:58:12.275646 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:12.275618    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1fba839-cxn8c\" (UID: \"7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fba839-cxn8c"
Nov 03 20:58:12.277359 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:12.277336    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1fba839-cxn8c\" (UID: \"7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fba839-cxn8c"
Nov 03 20:58:12.288004 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:12.287975    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc6pp\" (UniqueName: \"kubernetes.io/projected/7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6-kube-api-access-cc6pp\") pod \"maas-api-revoke-sub-f1fba839-cxn8c\" (UID: \"7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fba839-cxn8c"
Nov 03 20:58:12.427537 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:12.427452    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fba839-cxn8c"
Nov 03 20:58:12.705955 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:12.705928    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1fba839-cxn8c"]
Nov 03 20:58:12.708598 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:58:12.708572    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b7d4627_fe8b_4bd4_a6a8_49c3f9b149c6.slice/crio-60a165652b417bc02890a090e20c8b0d1ae9b5c9fff68b2ea35294e845ef8b11 WatchSource:0}: Error finding container 60a165652b417bc02890a090e20c8b0d1ae9b5c9fff68b2ea35294e845ef8b11: Status 404 returned error can't find the container with id 60a165652b417bc02890a090e20c8b0d1ae9b5c9fff68b2ea35294e845ef8b11
Nov 03 20:58:13.540666 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:13.540633    5591 generic.go:356] "Generic (PLEG): container finished" podID="7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6" containerID="d20bf423f5ec130bff23a1e070396922c71970d35a4405672956312523dea68a" exitCode=0
Nov 03 20:58:13.541099 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:13.540720    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fba839-cxn8c" event={"ID":"7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6","Type":"ContainerDied","Data":"d20bf423f5ec130bff23a1e070396922c71970d35a4405672956312523dea68a"}
Nov 03 20:58:13.541099 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:13.540777    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fba839-cxn8c" event={"ID":"7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6","Type":"ContainerStarted","Data":"60a165652b417bc02890a090e20c8b0d1ae9b5c9fff68b2ea35294e845ef8b11"}
Nov 03 20:58:14.686846 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:14.686819    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fba839-cxn8c"
Nov 03 20:58:14.798897 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:14.798797    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6-maas-api-openshift-ca\") pod \"7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6\" (UID: \"7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6\") "
Nov 03 20:58:14.798897 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:14.798838    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6-kube-api-access-cc6pp\" (UniqueName: \"kubernetes.io/projected/7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6-kube-api-access-cc6pp\") pod \"7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6\" (UID: \"7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6\") "
Nov 03 20:58:14.798897 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:14.798872    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6-maas-api-xks-ca\") pod \"7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6\" (UID: \"7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6\") "
Nov 03 20:58:14.799268 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:14.799176    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6-maas-api-openshift-ca" pod "7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6" (UID: "7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:58:14.799390 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:14.799343    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:14.801244 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:14.801220    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6-maas-api-xks-ca" pod "7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6" (UID: "7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:58:14.801357 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:14.801338    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6-kube-api-access-cc6pp" pod "7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6" (UID: "7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6"). InnerVolumeSpecName "kube-api-access-cc6pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:14.900117 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:14.900072    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc6pp\" (UniqueName: \"kubernetes.io/projected/7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6-kube-api-access-cc6pp\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:14.900117 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:14.900112    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:15.553385 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:15.553338    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fba839-cxn8c" event={"ID":"7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6","Type":"ContainerDied","Data":"60a165652b417bc02890a090e20c8b0d1ae9b5c9fff68b2ea35294e845ef8b11"}
Nov 03 20:58:15.553385 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:15.553382    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1fba839-cxn8c"
Nov 03 20:58:15.553615 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:15.553395    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60a165652b417bc02890a090e20c8b0d1ae9b5c9fff68b2ea35294e845ef8b11"
Nov 03 20:58:23.825816 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:23.825778    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:58:28.723990 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:58:28.723946    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:58:29.786098 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:29.786052    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k"]
Nov 03 20:58:29.803792 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:29.803755    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x"]
Nov 03 20:58:32.357744 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:32.357705    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:58:33.022723 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:33.022683    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhjqf"]
Nov 03 20:58:33.025346 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:33.025317    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-lhjqf"]
Nov 03 20:58:33.348405 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:33.348328    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f6e1af8-cfd9-45e6-82ac-3887872b8406" path="/var/lib/kubelet/pods/6f6e1af8-cfd9-45e6-82ac-3887872b8406/volumes"
Nov 03 20:58:34.787413 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:34.787339    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k" podUID="749a27eb-0c3d-4789-908a-180c7bb01e58" containerName="payload-processing" containerID="cri-o://de5ed29a6fb2f912e39ab4a612d733b117c5b8aa8815023e587a17de7eb79082" gracePeriod=25
Nov 03 20:58:34.805084 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:34.805041    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x" podUID="0f21c382-40db-4224-b589-a52a37298a4e" containerName="payload-pre-processing" containerID="cri-o://b4542a41d40b2a969ebb6cec8bb932954d39795d90286b7ab177b7ab0f329109" gracePeriod=25
Nov 03 20:58:38.377734 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:38.377694    5591 scope.go:122] "RemoveContainer" containerID="00e8aa151a86a579b164803bfb5d9c3d395fcc085370b814bf38163d68866ef4"
Nov 03 20:58:38.387516 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:38.387489    5591 scope.go:122] "RemoveContainer" containerID="3773c302c7b5fb7b80ff9f047fecb2ba24a607405b325c6d896508a9d335f53b"
Nov 03 20:58:43.025145 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:43.025058    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-89rl2"]
Nov 03 20:58:43.028286 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:43.028254    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-89rl2"]
Nov 03 20:58:43.220841 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:43.220800    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k" podUID="749a27eb-0c3d-4789-908a-180c7bb01e58" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:58:43.221052 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:43.220808    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x" podUID="0f21c382-40db-4224-b589-a52a37298a4e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:58:43.347961 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:43.347876    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="033b1579-4ddf-4680-8c72-6bd296480c9e" path="/var/lib/kubelet/pods/033b1579-4ddf-4680-8c72-6bd296480c9e/volumes"
Nov 03 20:58:53.221621 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:53.221576    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k" podUID="749a27eb-0c3d-4789-908a-180c7bb01e58" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:58:53.222066 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:53.221647    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x" podUID="0f21c382-40db-4224-b589-a52a37298a4e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 20:58:55.059349 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.059278    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k"
Nov 03 20:58:55.094253 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.094224    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x"
Nov 03 20:58:55.103792 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.103758    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/749a27eb-0c3d-4789-908a-180c7bb01e58-kube-api-access-nvhqf\" (UniqueName: \"kubernetes.io/projected/749a27eb-0c3d-4789-908a-180c7bb01e58-kube-api-access-nvhqf\") pod \"749a27eb-0c3d-4789-908a-180c7bb01e58\" (UID: \"749a27eb-0c3d-4789-908a-180c7bb01e58\") "
Nov 03 20:58:55.103966 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.103857    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/749a27eb-0c3d-4789-908a-180c7bb01e58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749a27eb-0c3d-4789-908a-180c7bb01e58-plugins-config-volume\") pod \"749a27eb-0c3d-4789-908a-180c7bb01e58\" (UID: \"749a27eb-0c3d-4789-908a-180c7bb01e58\") "
Nov 03 20:58:55.104287 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.104261    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/749a27eb-0c3d-4789-908a-180c7bb01e58-plugins-config-volume" pod "749a27eb-0c3d-4789-908a-180c7bb01e58" (UID: "749a27eb-0c3d-4789-908a-180c7bb01e58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:58:55.106088 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.106065    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/749a27eb-0c3d-4789-908a-180c7bb01e58-kube-api-access-nvhqf" pod "749a27eb-0c3d-4789-908a-180c7bb01e58" (UID: "749a27eb-0c3d-4789-908a-180c7bb01e58"). InnerVolumeSpecName "kube-api-access-nvhqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:55.204904 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.204866    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f21c382-40db-4224-b589-a52a37298a4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f21c382-40db-4224-b589-a52a37298a4e-plugins-config-volume\") pod \"0f21c382-40db-4224-b589-a52a37298a4e\" (UID: \"0f21c382-40db-4224-b589-a52a37298a4e\") "
Nov 03 20:58:55.205119 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.204925    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f21c382-40db-4224-b589-a52a37298a4e-kube-api-access-k2fcf\" (UniqueName: \"kubernetes.io/projected/0f21c382-40db-4224-b589-a52a37298a4e-kube-api-access-k2fcf\") pod \"0f21c382-40db-4224-b589-a52a37298a4e\" (UID: \"0f21c382-40db-4224-b589-a52a37298a4e\") "
Nov 03 20:58:55.205234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.205154    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749a27eb-0c3d-4789-908a-180c7bb01e58-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:55.205234 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.205165    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvhqf\" (UniqueName: \"kubernetes.io/projected/749a27eb-0c3d-4789-908a-180c7bb01e58-kube-api-access-nvhqf\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:55.205365 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.205338    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f21c382-40db-4224-b589-a52a37298a4e-plugins-config-volume" pod "0f21c382-40db-4224-b589-a52a37298a4e" (UID: "0f21c382-40db-4224-b589-a52a37298a4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:58:55.207356 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.207333    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f21c382-40db-4224-b589-a52a37298a4e-kube-api-access-k2fcf" pod "0f21c382-40db-4224-b589-a52a37298a4e" (UID: "0f21c382-40db-4224-b589-a52a37298a4e"). InnerVolumeSpecName "kube-api-access-k2fcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:58:55.305868 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.305830    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2fcf\" (UniqueName: \"kubernetes.io/projected/0f21c382-40db-4224-b589-a52a37298a4e-kube-api-access-k2fcf\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:55.305868 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.305864    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f21c382-40db-4224-b589-a52a37298a4e-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:58:55.786065 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.786026    5591 generic.go:356] "Generic (PLEG): container finished" podID="0f21c382-40db-4224-b589-a52a37298a4e" containerID="b4542a41d40b2a969ebb6cec8bb932954d39795d90286b7ab177b7ab0f329109" exitCode=0
Nov 03 20:58:55.786317 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.786075    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x"
Nov 03 20:58:55.786317 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.786114    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x" event={"ID":"0f21c382-40db-4224-b589-a52a37298a4e","Type":"ContainerDied","Data":"b4542a41d40b2a969ebb6cec8bb932954d39795d90286b7ab177b7ab0f329109"}
Nov 03 20:58:55.786317 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.786174    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x" event={"ID":"0f21c382-40db-4224-b589-a52a37298a4e","Type":"ContainerDied","Data":"53041918be98b97c7d16113ef24ec143871e3914fc5bac2bfed7a68c927eef41"}
Nov 03 20:58:55.786317 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.786219    5591 scope.go:122] "RemoveContainer" containerID="b4542a41d40b2a969ebb6cec8bb932954d39795d90286b7ab177b7ab0f329109"
Nov 03 20:58:55.787582 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.787561    5591 generic.go:356] "Generic (PLEG): container finished" podID="749a27eb-0c3d-4789-908a-180c7bb01e58" containerID="de5ed29a6fb2f912e39ab4a612d733b117c5b8aa8815023e587a17de7eb79082" exitCode=0
Nov 03 20:58:55.787711 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.787597    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k" event={"ID":"749a27eb-0c3d-4789-908a-180c7bb01e58","Type":"ContainerDied","Data":"de5ed29a6fb2f912e39ab4a612d733b117c5b8aa8815023e587a17de7eb79082"}
Nov 03 20:58:55.787711 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.787615    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k"
Nov 03 20:58:55.787711 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.787641    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k" event={"ID":"749a27eb-0c3d-4789-908a-180c7bb01e58","Type":"ContainerDied","Data":"f4f4a339e275ee5d7763a9f20bcf612841a54d499c615e17a7e33b930be9d835"}
Nov 03 20:58:55.795863 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.795846    5591 scope.go:122] "RemoveContainer" containerID="b4542a41d40b2a969ebb6cec8bb932954d39795d90286b7ab177b7ab0f329109"
Nov 03 20:58:55.796139 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:58:55.796111    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4542a41d40b2a969ebb6cec8bb932954d39795d90286b7ab177b7ab0f329109\": container with ID starting with b4542a41d40b2a969ebb6cec8bb932954d39795d90286b7ab177b7ab0f329109 not found: ID does not exist" containerID="b4542a41d40b2a969ebb6cec8bb932954d39795d90286b7ab177b7ab0f329109"
Nov 03 20:58:55.796254 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.796146    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4542a41d40b2a969ebb6cec8bb932954d39795d90286b7ab177b7ab0f329109"} err="failed to get container status \"b4542a41d40b2a969ebb6cec8bb932954d39795d90286b7ab177b7ab0f329109\": rpc error: code = NotFound desc = could not find container \"b4542a41d40b2a969ebb6cec8bb932954d39795d90286b7ab177b7ab0f329109\": container with ID starting with b4542a41d40b2a969ebb6cec8bb932954d39795d90286b7ab177b7ab0f329109 not found: ID does not exist"
Nov 03 20:58:55.796254 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.796168    5591 scope.go:122] "RemoveContainer" containerID="de5ed29a6fb2f912e39ab4a612d733b117c5b8aa8815023e587a17de7eb79082"
Nov 03 20:58:55.805093 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.805065    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x"]
Nov 03 20:58:55.809232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.809189    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f49qsx6x"]
Nov 03 20:58:55.810811 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.810786    5591 scope.go:122] "RemoveContainer" containerID="de5ed29a6fb2f912e39ab4a612d733b117c5b8aa8815023e587a17de7eb79082"
Nov 03 20:58:55.811150 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:58:55.811125    5591 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de5ed29a6fb2f912e39ab4a612d733b117c5b8aa8815023e587a17de7eb79082\": container with ID starting with de5ed29a6fb2f912e39ab4a612d733b117c5b8aa8815023e587a17de7eb79082 not found: ID does not exist" containerID="de5ed29a6fb2f912e39ab4a612d733b117c5b8aa8815023e587a17de7eb79082"
Nov 03 20:58:55.811226 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.811158    5591 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5ed29a6fb2f912e39ab4a612d733b117c5b8aa8815023e587a17de7eb79082"} err="failed to get container status \"de5ed29a6fb2f912e39ab4a612d733b117c5b8aa8815023e587a17de7eb79082\": rpc error: code = NotFound desc = could not find container \"de5ed29a6fb2f912e39ab4a612d733b117c5b8aa8815023e587a17de7eb79082\": container with ID starting with de5ed29a6fb2f912e39ab4a612d733b117c5b8aa8815023e587a17de7eb79082 not found: ID does not exist"
Nov 03 20:58:55.820867 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.820842    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k"]
Nov 03 20:58:55.827267 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:55.827242    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d4b5l4k"]
Nov 03 20:58:57.347367 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:57.347329    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f21c382-40db-4224-b589-a52a37298a4e" path="/var/lib/kubelet/pods/0f21c382-40db-4224-b589-a52a37298a4e/volumes"
Nov 03 20:58:57.347785 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:58:57.347718    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="749a27eb-0c3d-4789-908a-180c7bb01e58" path="/var/lib/kubelet/pods/749a27eb-0c3d-4789-908a-180c7bb01e58/volumes"
Nov 03 20:58:58.723872 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:58:58.723833    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:59:02.163158 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.163121    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf"]
Nov 03 20:59:02.163697 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.163599    5591 cpu_manager.go:414] "containerMap: removing container" podUID="749a27eb-0c3d-4789-908a-180c7bb01e58" containerName="payload-processing"
Nov 03 20:59:02.163697 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.163611    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="749a27eb-0c3d-4789-908a-180c7bb01e58" containerName="payload-processing"
Nov 03 20:59:02.163697 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.163629    5591 cpu_manager.go:414] "containerMap: removing container" podUID="7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6" containerName="revoke-keys"
Nov 03 20:59:02.163697 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.163634    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6" containerName="revoke-keys"
Nov 03 20:59:02.163697 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.163642    5591 cpu_manager.go:414] "containerMap: removing container" podUID="0f21c382-40db-4224-b589-a52a37298a4e" containerName="payload-pre-processing"
Nov 03 20:59:02.163697 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.163648    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f21c382-40db-4224-b589-a52a37298a4e" containerName="payload-pre-processing"
Nov 03 20:59:02.163939 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.163718    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="749a27eb-0c3d-4789-908a-180c7bb01e58" containerName="payload-processing"
Nov 03 20:59:02.163939 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.163727    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6" containerName="revoke-keys"
Nov 03 20:59:02.163939 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.163735    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f21c382-40db-4224-b589-a52a37298a4e" containerName="payload-pre-processing"
Nov 03 20:59:02.168251 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.168232    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf"
Nov 03 20:59:02.178255 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.178231    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-4ca23e\""
Nov 03 20:59:02.178619 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.178569    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf"]
Nov 03 20:59:02.179324 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.178725    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-963f71df-dockercfg-bx5xt\""
Nov 03 20:59:02.180972 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.180946    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp"]
Nov 03 20:59:02.185282 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.185255    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp"
Nov 03 20:59:02.197830 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.197801    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp"]
Nov 03 20:59:02.273141 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.273099    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxdd8\" (UniqueName: \"kubernetes.io/projected/c00b1cdb-7f58-4360-b89b-714278f71d61-kube-api-access-xxdd8\") pod \"payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp\" (UID: \"c00b1cdb-7f58-4360-b89b-714278f71d61\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp"
Nov 03 20:59:02.273431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.273188    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00b1cdb-7f58-4360-b89b-714278f71d61-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp\" (UID: \"c00b1cdb-7f58-4360-b89b-714278f71d61\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp"
Nov 03 20:59:02.273431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.273260    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qhq2\" (UniqueName: \"kubernetes.io/projected/50f2ae70-26b7-40d6-bd5d-343d002b63ff-kube-api-access-9qhq2\") pod \"payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf\" (UID: \"50f2ae70-26b7-40d6-bd5d-343d002b63ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf"
Nov 03 20:59:02.273431 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.273323    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f2ae70-26b7-40d6-bd5d-343d002b63ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf\" (UID: \"50f2ae70-26b7-40d6-bd5d-343d002b63ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf"
Nov 03 20:59:02.374542 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.374507    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f2ae70-26b7-40d6-bd5d-343d002b63ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf\" (UID: \"50f2ae70-26b7-40d6-bd5d-343d002b63ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf"
Nov 03 20:59:02.374778 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.374639    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxdd8\" (UniqueName: \"kubernetes.io/projected/c00b1cdb-7f58-4360-b89b-714278f71d61-kube-api-access-xxdd8\") pod \"payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp\" (UID: \"c00b1cdb-7f58-4360-b89b-714278f71d61\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp"
Nov 03 20:59:02.374778 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.374700    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00b1cdb-7f58-4360-b89b-714278f71d61-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp\" (UID: \"c00b1cdb-7f58-4360-b89b-714278f71d61\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp"
Nov 03 20:59:02.374983 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.374942    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qhq2\" (UniqueName: \"kubernetes.io/projected/50f2ae70-26b7-40d6-bd5d-343d002b63ff-kube-api-access-9qhq2\") pod \"payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf\" (UID: \"50f2ae70-26b7-40d6-bd5d-343d002b63ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf"
Nov 03 20:59:02.375339 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.375317    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f2ae70-26b7-40d6-bd5d-343d002b63ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf\" (UID: \"50f2ae70-26b7-40d6-bd5d-343d002b63ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf"
Nov 03 20:59:02.375562 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.375540    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00b1cdb-7f58-4360-b89b-714278f71d61-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp\" (UID: \"c00b1cdb-7f58-4360-b89b-714278f71d61\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp"
Nov 03 20:59:02.386979 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.386955    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qhq2\" (UniqueName: \"kubernetes.io/projected/50f2ae70-26b7-40d6-bd5d-343d002b63ff-kube-api-access-9qhq2\") pod \"payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf\" (UID: \"50f2ae70-26b7-40d6-bd5d-343d002b63ff\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf"
Nov 03 20:59:02.387085 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.386987    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxdd8\" (UniqueName: \"kubernetes.io/projected/c00b1cdb-7f58-4360-b89b-714278f71d61-kube-api-access-xxdd8\") pod \"payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp\" (UID: \"c00b1cdb-7f58-4360-b89b-714278f71d61\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp"
Nov 03 20:59:02.484021 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.483923    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf"
Nov 03 20:59:02.499307 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.499274    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp"
Nov 03 20:59:02.769026 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.768990    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf"]
Nov 03 20:59:02.773727 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:59:02.773690    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f2ae70_26b7_40d6_bd5d_343d002b63ff.slice/crio-54d3b16ba2e9e9c1ec5cc37d1d11d53bf68fdacf0d93ff8055fb71303bd491cc WatchSource:0}: Error finding container 54d3b16ba2e9e9c1ec5cc37d1d11d53bf68fdacf0d93ff8055fb71303bd491cc: Status 404 returned error can't find the container with id 54d3b16ba2e9e9c1ec5cc37d1d11d53bf68fdacf0d93ff8055fb71303bd491cc
Nov 03 20:59:02.788537 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.788451    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp"]
Nov 03 20:59:02.791165 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:59:02.791138    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00b1cdb_7f58_4360_b89b_714278f71d61.slice/crio-ab0af4617a2d424a6358e3e9a3443733dd91b872008b13c6898cf4dea98a0143 WatchSource:0}: Error finding container ab0af4617a2d424a6358e3e9a3443733dd91b872008b13c6898cf4dea98a0143: Status 404 returned error can't find the container with id ab0af4617a2d424a6358e3e9a3443733dd91b872008b13c6898cf4dea98a0143
Nov 03 20:59:02.828911 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.828792    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp" event={"ID":"c00b1cdb-7f58-4360-b89b-714278f71d61","Type":"ContainerStarted","Data":"ab0af4617a2d424a6358e3e9a3443733dd91b872008b13c6898cf4dea98a0143"}
Nov 03 20:59:02.831332 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:02.831300    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf" event={"ID":"50f2ae70-26b7-40d6-bd5d-343d002b63ff","Type":"ContainerStarted","Data":"54d3b16ba2e9e9c1ec5cc37d1d11d53bf68fdacf0d93ff8055fb71303bd491cc"}
Nov 03 20:59:03.838731 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:03.838690    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp" event={"ID":"c00b1cdb-7f58-4360-b89b-714278f71d61","Type":"ContainerStarted","Data":"e43a449c61345d7452b74822b2c8e4bc1c70e8807ffd865a8e2c52f474af7140"}
Nov 03 20:59:03.839306 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:03.838787    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp"
Nov 03 20:59:03.840453 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:03.840428    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf" event={"ID":"50f2ae70-26b7-40d6-bd5d-343d002b63ff","Type":"ContainerStarted","Data":"b1ced1db8847039849eee5f4f3c528334782e1a0bd71c91aa1272086fd6e2b09"}
Nov 03 20:59:03.840580 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:03.840563    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf"
Nov 03 20:59:03.859493 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:03.859438    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp" podStartSLOduration=1.8594233180000002 podStartE2EDuration="1.859423318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:03.858007028 +0000 UTC m=+4892.630300045" watchObservedRunningTime="2026-11-03 20:59:03.859423318 +0000 UTC m=+4892.631716335"
Nov 03 20:59:03.878048 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:03.877985    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf" podStartSLOduration=1.877971174 podStartE2EDuration="1.877971174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:03.875492957 +0000 UTC m=+4892.647785974" watchObservedRunningTime="2026-11-03 20:59:03.877971174 +0000 UTC m=+4892.650264192"
Nov 03 20:59:04.419783 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:04.419735    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:59:07.447402 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:07.447363    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:59:10.337407 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:10.337370    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2"]
Nov 03 20:59:10.341572 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:10.341542    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2"
Nov 03 20:59:10.347459 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:10.347412    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 20:59:10.348420 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:10.348379    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 20:59:10.348586 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:10.348490    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 20:59:10.348711 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:10.348682    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 20:59:10.350524 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:10.350231    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2"]
Nov 03 20:59:10.460849 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:10.460793    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr98q\" (UniqueName: \"kubernetes.io/projected/49525ed7-3d7f-4280-85e9-0b48ddd222ad-kube-api-access-pr98q\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2\" (UID: \"49525ed7-3d7f-4280-85e9-0b48ddd222ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2"
Nov 03 20:59:10.461085 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:10.460934    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49525ed7-3d7f-4280-85e9-0b48ddd222ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2\" (UID: \"49525ed7-3d7f-4280-85e9-0b48ddd222ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2"
Nov 03 20:59:10.461245 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:10.461189    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49525ed7-3d7f-4280-85e9-0b48ddd222ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2\" (UID: \"49525ed7-3d7f-4280-85e9-0b48ddd222ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2"
Nov 03 20:59:10.562699 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:10.562666    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49525ed7-3d7f-4280-85e9-0b48ddd222ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2\" (UID: \"49525ed7-3d7f-4280-85e9-0b48ddd222ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2"
Nov 03 20:59:10.562922 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:10.562765    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr98q\" (UniqueName: \"kubernetes.io/projected/49525ed7-3d7f-4280-85e9-0b48ddd222ad-kube-api-access-pr98q\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2\" (UID: \"49525ed7-3d7f-4280-85e9-0b48ddd222ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2"
Nov 03 20:59:10.562922 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:10.562792    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49525ed7-3d7f-4280-85e9-0b48ddd222ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2\" (UID: \"49525ed7-3d7f-4280-85e9-0b48ddd222ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2"
Nov 03 20:59:10.563535 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:10.563471    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49525ed7-3d7f-4280-85e9-0b48ddd222ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2\" (UID: \"49525ed7-3d7f-4280-85e9-0b48ddd222ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2"
Nov 03 20:59:10.565246 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:10.565197    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49525ed7-3d7f-4280-85e9-0b48ddd222ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2\" (UID: \"49525ed7-3d7f-4280-85e9-0b48ddd222ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2"
Nov 03 20:59:10.575936 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:10.575916    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr98q\" (UniqueName: \"kubernetes.io/projected/49525ed7-3d7f-4280-85e9-0b48ddd222ad-kube-api-access-pr98q\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2\" (UID: \"49525ed7-3d7f-4280-85e9-0b48ddd222ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2"
Nov 03 20:59:10.656967 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:10.656873    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2"
Nov 03 20:59:10.937438 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:10.937413    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2"]
Nov 03 20:59:10.939820 ip-10-0-143-206 kubenswrapper[5591]: W1103 20:59:10.939793    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49525ed7_3d7f_4280_85e9_0b48ddd222ad.slice/crio-60f47b41f74100065e68a8f0c83a45974be4a73b6891df7b57c51f5098755ed5 WatchSource:0}: Error finding container 60f47b41f74100065e68a8f0c83a45974be4a73b6891df7b57c51f5098755ed5: Status 404 returned error can't find the container with id 60f47b41f74100065e68a8f0c83a45974be4a73b6891df7b57c51f5098755ed5
Nov 03 20:59:11.021776 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:11.021737    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5b62q"]
Nov 03 20:59:11.024040 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:11.024000    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5b62q"]
Nov 03 20:59:11.347906 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:11.347870    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c8a1c63-e110-435b-9b93-f6417a37d003" path="/var/lib/kubelet/pods/5c8a1c63-e110-435b-9b93-f6417a37d003/volumes"
Nov 03 20:59:11.700152 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:59:11.700121    5591 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49525ed7_3d7f_4280_85e9_0b48ddd222ad.slice/crio-ce5754ac4f9fbfb0c90571d6c04fc2082657e608e09e715cbb921e1dc040f364.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 20:59:11.887815 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:11.887773    5591 generic.go:356] "Generic (PLEG): container finished" podID="49525ed7-3d7f-4280-85e9-0b48ddd222ad" containerID="ce5754ac4f9fbfb0c90571d6c04fc2082657e608e09e715cbb921e1dc040f364" exitCode=0
Nov 03 20:59:11.888002 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:11.887822    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2" event={"ID":"49525ed7-3d7f-4280-85e9-0b48ddd222ad","Type":"ContainerDied","Data":"ce5754ac4f9fbfb0c90571d6c04fc2082657e608e09e715cbb921e1dc040f364"}
Nov 03 20:59:11.888002 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:11.887870    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2" event={"ID":"49525ed7-3d7f-4280-85e9-0b48ddd222ad","Type":"ContainerStarted","Data":"60f47b41f74100065e68a8f0c83a45974be4a73b6891df7b57c51f5098755ed5"}
Nov 03 20:59:13.032001 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:13.031970    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2"
Nov 03 20:59:13.187086 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:13.186993    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49525ed7-3d7f-4280-85e9-0b48ddd222ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49525ed7-3d7f-4280-85e9-0b48ddd222ad-maas-api-xks-ca\") pod \"49525ed7-3d7f-4280-85e9-0b48ddd222ad\" (UID: \"49525ed7-3d7f-4280-85e9-0b48ddd222ad\") "
Nov 03 20:59:13.187294 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:13.187144    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49525ed7-3d7f-4280-85e9-0b48ddd222ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49525ed7-3d7f-4280-85e9-0b48ddd222ad-maas-api-openshift-ca\") pod \"49525ed7-3d7f-4280-85e9-0b48ddd222ad\" (UID: \"49525ed7-3d7f-4280-85e9-0b48ddd222ad\") "
Nov 03 20:59:13.187294 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:13.187186    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49525ed7-3d7f-4280-85e9-0b48ddd222ad-kube-api-access-pr98q\" (UniqueName: \"kubernetes.io/projected/49525ed7-3d7f-4280-85e9-0b48ddd222ad-kube-api-access-pr98q\") pod \"49525ed7-3d7f-4280-85e9-0b48ddd222ad\" (UID: \"49525ed7-3d7f-4280-85e9-0b48ddd222ad\") "
Nov 03 20:59:13.187577 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:13.187547    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49525ed7-3d7f-4280-85e9-0b48ddd222ad-maas-api-openshift-ca" pod "49525ed7-3d7f-4280-85e9-0b48ddd222ad" (UID: "49525ed7-3d7f-4280-85e9-0b48ddd222ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:59:13.189503 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:13.189475    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49525ed7-3d7f-4280-85e9-0b48ddd222ad-maas-api-xks-ca" pod "49525ed7-3d7f-4280-85e9-0b48ddd222ad" (UID: "49525ed7-3d7f-4280-85e9-0b48ddd222ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 20:59:13.189601 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:13.189556    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49525ed7-3d7f-4280-85e9-0b48ddd222ad-kube-api-access-pr98q" pod "49525ed7-3d7f-4280-85e9-0b48ddd222ad" (UID: "49525ed7-3d7f-4280-85e9-0b48ddd222ad"). InnerVolumeSpecName "kube-api-access-pr98q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:13.288557 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:13.288518    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49525ed7-3d7f-4280-85e9-0b48ddd222ad-maas-api-openshift-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:13.288557 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:13.288559    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr98q\" (UniqueName: \"kubernetes.io/projected/49525ed7-3d7f-4280-85e9-0b48ddd222ad-kube-api-access-pr98q\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:13.288775 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:13.288576    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49525ed7-3d7f-4280-85e9-0b48ddd222ad-maas-api-xks-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:13.900291 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:13.900255    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2"
Nov 03 20:59:13.900291 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:13.900271    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2" event={"ID":"49525ed7-3d7f-4280-85e9-0b48ddd222ad","Type":"ContainerDied","Data":"60f47b41f74100065e68a8f0c83a45974be4a73b6891df7b57c51f5098755ed5"}
Nov 03 20:59:13.900613 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:13.900324    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60f47b41f74100065e68a8f0c83a45974be4a73b6891df7b57c51f5098755ed5"
Nov 03 20:59:14.849299 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:14.849270    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf"
Nov 03 20:59:14.849821 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:14.849338    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp"
Nov 03 20:59:20.461634 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:20.461597    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp"]
Nov 03 20:59:20.476563 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:20.476525    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf"]
Nov 03 20:59:21.407872 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:21.407837    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2"]
Nov 03 20:59:21.410023 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:21.410000    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4ca23e-md4k2"]
Nov 03 20:59:21.669123 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:21.669033    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"]
Nov 03 20:59:21.669533 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:21.669337    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw" podUID="f10a9b94-0e91-422a-a65d-4e58c0452d6e" containerName="istio-proxy" containerID="cri-o://38946927e31d006fc8e5dd45cd259e03b11f6e2457a13dd57ea9f8379fe3a551" gracePeriod=30
Nov 03 20:59:22.708048 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:22.708007    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:59:22.968919 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:22.968822    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8njpn"]
Nov 03 20:59:22.973469 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:22.973438    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8njpn"]
Nov 03 20:59:23.348107 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:23.348067    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40200261-61e0-4f57-9d8b-2b16da0ede93" path="/var/lib/kubelet/pods/40200261-61e0-4f57-9d8b-2b16da0ede93/volumes"
Nov 03 20:59:23.348598 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:23.348580    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49525ed7-3d7f-4280-85e9-0b48ddd222ad" path="/var/lib/kubelet/pods/49525ed7-3d7f-4280-85e9-0b48ddd222ad/volumes"
Nov 03 20:59:24.819131 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:24.819088    5591 patch_prober.go:29] interesting pod/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 20:59:24.819638 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:24.819161    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw" podUID="f10a9b94-0e91-422a-a65d-4e58c0452d6e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 20:59:25.462916 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:25.462837    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp" podUID="c00b1cdb-7f58-4360-b89b-714278f71d61" containerName="payload-processing" containerID="cri-o://e43a449c61345d7452b74822b2c8e4bc1c70e8807ffd865a8e2c52f474af7140" gracePeriod=25
Nov 03 20:59:25.477370 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:25.477299    5591 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf" podUID="50f2ae70-26b7-40d6-bd5d-343d002b63ff" containerName="payload-pre-processing" containerID="cri-o://b1ced1db8847039849eee5f4f3c528334782e1a0bd71c91aa1272086fd6e2b09" gracePeriod=25
Nov 03 20:59:26.053572 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:26.043841    5591 generic.go:356] "Generic (PLEG): container finished" podID="c00b1cdb-7f58-4360-b89b-714278f71d61" containerID="e43a449c61345d7452b74822b2c8e4bc1c70e8807ffd865a8e2c52f474af7140" exitCode=0
Nov 03 20:59:26.053572 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:26.044021    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp" event={"ID":"c00b1cdb-7f58-4360-b89b-714278f71d61","Type":"ContainerDied","Data":"e43a449c61345d7452b74822b2c8e4bc1c70e8807ffd865a8e2c52f474af7140"}
Nov 03 20:59:26.053572 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:26.044079    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp" event={"ID":"c00b1cdb-7f58-4360-b89b-714278f71d61","Type":"ContainerDied","Data":"ab0af4617a2d424a6358e3e9a3443733dd91b872008b13c6898cf4dea98a0143"}
Nov 03 20:59:26.053572 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:26.044104    5591 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab0af4617a2d424a6358e3e9a3443733dd91b872008b13c6898cf4dea98a0143"
Nov 03 20:59:26.066530 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:26.066499    5591 generic.go:356] "Generic (PLEG): container finished" podID="50f2ae70-26b7-40d6-bd5d-343d002b63ff" containerID="b1ced1db8847039849eee5f4f3c528334782e1a0bd71c91aa1272086fd6e2b09" exitCode=0
Nov 03 20:59:26.066814 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:26.066598    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf" event={"ID":"50f2ae70-26b7-40d6-bd5d-343d002b63ff","Type":"ContainerDied","Data":"b1ced1db8847039849eee5f4f3c528334782e1a0bd71c91aa1272086fd6e2b09"}
Nov 03 20:59:26.070103 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:26.070081    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp"
Nov 03 20:59:26.175295 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:26.175262    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf"
Nov 03 20:59:26.267842 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:26.267810    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c00b1cdb-7f58-4360-b89b-714278f71d61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00b1cdb-7f58-4360-b89b-714278f71d61-plugins-config-volume\") pod \"c00b1cdb-7f58-4360-b89b-714278f71d61\" (UID: \"c00b1cdb-7f58-4360-b89b-714278f71d61\") "
Nov 03 20:59:26.268021 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:26.267906    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c00b1cdb-7f58-4360-b89b-714278f71d61-kube-api-access-xxdd8\" (UniqueName: \"kubernetes.io/projected/c00b1cdb-7f58-4360-b89b-714278f71d61-kube-api-access-xxdd8\") pod \"c00b1cdb-7f58-4360-b89b-714278f71d61\" (UID: \"c00b1cdb-7f58-4360-b89b-714278f71d61\") "
Nov 03 20:59:26.268494 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:26.268471    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00b1cdb-7f58-4360-b89b-714278f71d61-plugins-config-volume" pod "c00b1cdb-7f58-4360-b89b-714278f71d61" (UID: "c00b1cdb-7f58-4360-b89b-714278f71d61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:59:26.270927 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:26.270898    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c00b1cdb-7f58-4360-b89b-714278f71d61-kube-api-access-xxdd8" pod "c00b1cdb-7f58-4360-b89b-714278f71d61" (UID: "c00b1cdb-7f58-4360-b89b-714278f71d61"). InnerVolumeSpecName "kube-api-access-xxdd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:26.368766 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:26.368726    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50f2ae70-26b7-40d6-bd5d-343d002b63ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f2ae70-26b7-40d6-bd5d-343d002b63ff-plugins-config-volume\") pod \"50f2ae70-26b7-40d6-bd5d-343d002b63ff\" (UID: \"50f2ae70-26b7-40d6-bd5d-343d002b63ff\") "
Nov 03 20:59:26.368965 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:26.368771    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50f2ae70-26b7-40d6-bd5d-343d002b63ff-kube-api-access-9qhq2\" (UniqueName: \"kubernetes.io/projected/50f2ae70-26b7-40d6-bd5d-343d002b63ff-kube-api-access-9qhq2\") pod \"50f2ae70-26b7-40d6-bd5d-343d002b63ff\" (UID: \"50f2ae70-26b7-40d6-bd5d-343d002b63ff\") "
Nov 03 20:59:26.369127 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:26.369109    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxdd8\" (UniqueName: \"kubernetes.io/projected/c00b1cdb-7f58-4360-b89b-714278f71d61-kube-api-access-xxdd8\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:26.369175 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:26.369136    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c00b1cdb-7f58-4360-b89b-714278f71d61-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:26.369243 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:26.369175    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f2ae70-26b7-40d6-bd5d-343d002b63ff-plugins-config-volume" pod "50f2ae70-26b7-40d6-bd5d-343d002b63ff" (UID: "50f2ae70-26b7-40d6-bd5d-343d002b63ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:59:26.371058 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:26.371035    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f2ae70-26b7-40d6-bd5d-343d002b63ff-kube-api-access-9qhq2" pod "50f2ae70-26b7-40d6-bd5d-343d002b63ff" (UID: "50f2ae70-26b7-40d6-bd5d-343d002b63ff"). InnerVolumeSpecName "kube-api-access-9qhq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:26.469969 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:26.469868    5591 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50f2ae70-26b7-40d6-bd5d-343d002b63ff-plugins-config-volume\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:26.469969 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:26.469908    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qhq2\" (UniqueName: \"kubernetes.io/projected/50f2ae70-26b7-40d6-bd5d-343d002b63ff-kube-api-access-9qhq2\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:26.788848 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:26.788318    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-mrwmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 20:59:26.788848 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:26.788382    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt" podUID="e14ca8f3-c80b-4c0b-9564-ef9734b86de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 20:59:27.013232 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.013089    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:59:27.088141 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.079957    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf" event={"ID":"50f2ae70-26b7-40d6-bd5d-343d002b63ff","Type":"ContainerDied","Data":"54d3b16ba2e9e9c1ec5cc37d1d11d53bf68fdacf0d93ff8055fb71303bd491cc"}
Nov 03 20:59:27.088141 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.080037    5591 scope.go:122] "RemoveContainer" containerID="b1ced1db8847039849eee5f4f3c528334782e1a0bd71c91aa1272086fd6e2b09"
Nov 03 20:59:27.088141 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.080230    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf"
Nov 03 20:59:27.090004 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.089969    5591 generic.go:356] "Generic (PLEG): container finished" podID="f10a9b94-0e91-422a-a65d-4e58c0452d6e" containerID="38946927e31d006fc8e5dd45cd259e03b11f6e2457a13dd57ea9f8379fe3a551" exitCode=0
Nov 03 20:59:27.090157 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.090142    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp"
Nov 03 20:59:27.091025 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.090923    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw" event={"ID":"f10a9b94-0e91-422a-a65d-4e58c0452d6e","Type":"ContainerDied","Data":"38946927e31d006fc8e5dd45cd259e03b11f6e2457a13dd57ea9f8379fe3a551"}
Nov 03 20:59:27.132456 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.132420    5591 scope.go:122] "RemoveContainer" containerID="da5f16364b96cf875494b75424d73b7d543ccbdedce10de148a1fca4e2f0f037"
Nov 03 20:59:27.143136 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.143112    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:59:27.148130 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.148108    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf"]
Nov 03 20:59:27.152827 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.152807    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4ca23e-56f497sbsf"]
Nov 03 20:59:27.163389 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.163368    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp"]
Nov 03 20:59:27.167616 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.167595    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4ca23e-54f8fc6d42w9pp"]
Nov 03 20:59:27.176640 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.176619    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-data\") pod \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") "
Nov 03 20:59:27.176749 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.176649    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-token\" (UniqueName: \"kubernetes.io/projected/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-token\") pod \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") "
Nov 03 20:59:27.176749 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.176690    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-envoy\") pod \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") "
Nov 03 20:59:27.176749 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.176723    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f10a9b94-0e91-422a-a65d-4e58c0452d6e-kube-api-access-xtdhq\" (UniqueName: \"kubernetes.io/projected/f10a9b94-0e91-422a-a65d-4e58c0452d6e-kube-api-access-xtdhq\") pod \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") "
Nov 03 20:59:27.176996 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.176765    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-workload-socket\") pod \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") "
Nov 03 20:59:27.176996 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.176813    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-credential-socket\") pod \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") "
Nov 03 20:59:27.176996 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.176858    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-podinfo\") pod \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") "
Nov 03 20:59:27.176996 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.176925    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-data" pod "f10a9b94-0e91-422a-a65d-4e58c0452d6e" (UID: "f10a9b94-0e91-422a-a65d-4e58c0452d6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:27.176996 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.176936    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istiod-ca-cert\") pod \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") "
Nov 03 20:59:27.176996 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.176973    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-workload-certs\") pod \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\" (UID: \"f10a9b94-0e91-422a-a65d-4e58c0452d6e\") "
Nov 03 20:59:27.177365 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.177145    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-credential-socket" pod "f10a9b94-0e91-422a-a65d-4e58c0452d6e" (UID: "f10a9b94-0e91-422a-a65d-4e58c0452d6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:27.177365 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.177332    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-workload-socket" pod "f10a9b94-0e91-422a-a65d-4e58c0452d6e" (UID: "f10a9b94-0e91-422a-a65d-4e58c0452d6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:27.177479 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.177361    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-workload-certs" pod "f10a9b94-0e91-422a-a65d-4e58c0452d6e" (UID: "f10a9b94-0e91-422a-a65d-4e58c0452d6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:27.177479 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.177398    5591 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-data\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:27.177479 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.177417    5591 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-credential-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:27.177821 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.177681    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istiod-ca-cert" pod "f10a9b94-0e91-422a-a65d-4e58c0452d6e" (UID: "f10a9b94-0e91-422a-a65d-4e58c0452d6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 20:59:27.179172 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.179148    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-envoy" pod "f10a9b94-0e91-422a-a65d-4e58c0452d6e" (UID: "f10a9b94-0e91-422a-a65d-4e58c0452d6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 20:59:27.179476 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.179454    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10a9b94-0e91-422a-a65d-4e58c0452d6e-kube-api-access-xtdhq" pod "f10a9b94-0e91-422a-a65d-4e58c0452d6e" (UID: "f10a9b94-0e91-422a-a65d-4e58c0452d6e"). InnerVolumeSpecName "kube-api-access-xtdhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:27.179919 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.179903    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-token" pod "f10a9b94-0e91-422a-a65d-4e58c0452d6e" (UID: "f10a9b94-0e91-422a-a65d-4e58c0452d6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 20:59:27.179984 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.179966    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-podinfo" pod "f10a9b94-0e91-422a-a65d-4e58c0452d6e" (UID: "f10a9b94-0e91-422a-a65d-4e58c0452d6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 20:59:27.278874 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.278842    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtdhq\" (UniqueName: \"kubernetes.io/projected/f10a9b94-0e91-422a-a65d-4e58c0452d6e-kube-api-access-xtdhq\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:27.278874 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.278873    5591 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-workload-socket\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:27.279090 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.278883    5591 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-podinfo\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:27.279090 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.278892    5591 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istiod-ca-cert\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:27.279090 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.278902    5591 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-workload-certs\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:27.279090 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.278910    5591 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-token\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:27.279090 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.278918    5591 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f10a9b94-0e91-422a-a65d-4e58c0452d6e-istio-envoy\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 20:59:27.355402 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.355369    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50f2ae70-26b7-40d6-bd5d-343d002b63ff" path="/var/lib/kubelet/pods/50f2ae70-26b7-40d6-bd5d-343d002b63ff/volumes"
Nov 03 20:59:27.355765 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:27.355753    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c00b1cdb-7f58-4360-b89b-714278f71d61" path="/var/lib/kubelet/pods/c00b1cdb-7f58-4360-b89b-714278f71d61/volumes"
Nov 03 20:59:28.096422 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:28.096393    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"
Nov 03 20:59:28.096422 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:28.096416    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw" event={"ID":"f10a9b94-0e91-422a-a65d-4e58c0452d6e","Type":"ContainerDied","Data":"82c37cde8f796407346336000e92070578cb20dfb4dfa433f83ecdbd7689e0cb"}
Nov 03 20:59:28.096986 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:28.096468    5591 scope.go:122] "RemoveContainer" containerID="38946927e31d006fc8e5dd45cd259e03b11f6e2457a13dd57ea9f8379fe3a551"
Nov 03 20:59:28.133521 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:28.133493    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"]
Nov 03 20:59:28.137935 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:28.137905    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4ca23e-openshift-default-55cd84869-5shmw"]
Nov 03 20:59:28.722675 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:59:28.722640    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:59:29.026553 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:29.026519    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5wng8"]
Nov 03 20:59:29.029236 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:29.029213    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5wng8"]
Nov 03 20:59:29.348100 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:29.348018    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1" path="/var/lib/kubelet/pods/7a1b65e5-c4b3-4d7d-9d3f-9330cc6e7eb1/volumes"
Nov 03 20:59:29.348522 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:29.348505    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f10a9b94-0e91-422a-a65d-4e58c0452d6e" path="/var/lib/kubelet/pods/f10a9b94-0e91-422a-a65d-4e58c0452d6e/volumes"
Nov 03 20:59:34.516421 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:34.516382    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 20:59:38.415173 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:38.415140    5591 scope.go:122] "RemoveContainer" containerID="335e8f0ae55e58cddcfb2445a950e1e366969e1d11d48ada739f61cc52da41fd"
Nov 03 20:59:38.425254 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:38.425228    5591 scope.go:122] "RemoveContainer" containerID="94433728f80269c8cff785a8fcf1ea81910608e39b850cbdbe94915004b14afa"
Nov 03 20:59:38.434075 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:38.434050    5591 scope.go:122] "RemoveContainer" containerID="d87ce0742efd9d18aa943b641c229509d01137f854351cf2402ef8237bec62d5"
Nov 03 20:59:44.023062 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:44.023025    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d5lq5"]
Nov 03 20:59:44.027963 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:44.027933    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-d5lq5"]
Nov 03 20:59:45.347661 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:45.347618    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1385f77b-5fde-4906-97ae-10b42a0dfb00" path="/var/lib/kubelet/pods/1385f77b-5fde-4906-97ae-10b42a0dfb00/volumes"
Nov 03 20:59:58.722517 ip-10-0-143-206 kubenswrapper[5591]: E1103 20:59:58.722475    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 20:59:59.024345 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:59.024312    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2w7p"]
Nov 03 20:59:59.028101 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:59.028070    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-q2w7p"]
Nov 03 20:59:59.347741 ip-10-0-143-206 kubenswrapper[5591]: I1103 20:59:59.347653    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40236cb7-8e9d-48e0-994d-bc1580d17fd9" path="/var/lib/kubelet/pods/40236cb7-8e9d-48e0-994d-bc1580d17fd9/volumes"
Nov 03 21:00:00.138980 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.135541    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-5b2m8"]
Nov 03 21:00:00.138980 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.136506    5591 cpu_manager.go:414] "containerMap: removing container" podUID="c00b1cdb-7f58-4360-b89b-714278f71d61" containerName="payload-processing"
Nov 03 21:00:00.138980 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.136529    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c00b1cdb-7f58-4360-b89b-714278f71d61" containerName="payload-processing"
Nov 03 21:00:00.138980 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.136641    5591 cpu_manager.go:414] "containerMap: removing container" podUID="f10a9b94-0e91-422a-a65d-4e58c0452d6e" containerName="istio-proxy"
Nov 03 21:00:00.138980 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.136649    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f10a9b94-0e91-422a-a65d-4e58c0452d6e" containerName="istio-proxy"
Nov 03 21:00:00.138980 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.136668    5591 cpu_manager.go:414] "containerMap: removing container" podUID="f10a9b94-0e91-422a-a65d-4e58c0452d6e" containerName="istio-proxy"
Nov 03 21:00:00.138980 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.136675    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f10a9b94-0e91-422a-a65d-4e58c0452d6e" containerName="istio-proxy"
Nov 03 21:00:00.138980 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.136686    5591 cpu_manager.go:414] "containerMap: removing container" podUID="49525ed7-3d7f-4280-85e9-0b48ddd222ad" containerName="revoke-keys"
Nov 03 21:00:00.138980 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.136693    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49525ed7-3d7f-4280-85e9-0b48ddd222ad" containerName="revoke-keys"
Nov 03 21:00:00.138980 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.136733    5591 cpu_manager.go:414] "containerMap: removing container" podUID="50f2ae70-26b7-40d6-bd5d-343d002b63ff" containerName="payload-pre-processing"
Nov 03 21:00:00.138980 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.136741    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f2ae70-26b7-40d6-bd5d-343d002b63ff" containerName="payload-pre-processing"
Nov 03 21:00:00.138980 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.136924    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f10a9b94-0e91-422a-a65d-4e58c0452d6e" containerName="istio-proxy"
Nov 03 21:00:00.138980 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.136945    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="f10a9b94-0e91-422a-a65d-4e58c0452d6e" containerName="istio-proxy"
Nov 03 21:00:00.138980 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.136954    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="49525ed7-3d7f-4280-85e9-0b48ddd222ad" containerName="revoke-keys"
Nov 03 21:00:00.138980 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.136972    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f2ae70-26b7-40d6-bd5d-343d002b63ff" containerName="payload-pre-processing"
Nov 03 21:00:00.138980 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.136988    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="c00b1cdb-7f58-4360-b89b-714278f71d61" containerName="payload-processing"
Nov 03 21:00:00.142776 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.142748    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-5b2m8"
Nov 03 21:00:00.146144 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.146118    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-5b2m8"]
Nov 03 21:00:00.146754 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.146727    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:00:00.147836 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.147805    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:00:00.147969 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.147842    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gqb8g\""
Nov 03 21:00:00.206483 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.206454    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5qjx\" (UniqueName: \"kubernetes.io/projected/61fe3d3b-5d3e-47b5-be4d-46ebd6f02914-kube-api-access-f5qjx\") pod \"maas-api-key-cleanup-29895660-5b2m8\" (UID: \"61fe3d3b-5d3e-47b5-be4d-46ebd6f02914\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-5b2m8"
Nov 03 21:00:00.206631 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.206497    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fe3d3b-5d3e-47b5-be4d-46ebd6f02914-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895660-5b2m8\" (UID: \"61fe3d3b-5d3e-47b5-be4d-46ebd6f02914\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-5b2m8"
Nov 03 21:00:00.307072 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.307036    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fe3d3b-5d3e-47b5-be4d-46ebd6f02914-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895660-5b2m8\" (UID: \"61fe3d3b-5d3e-47b5-be4d-46ebd6f02914\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-5b2m8"
Nov 03 21:00:00.307295 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.307164    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5qjx\" (UniqueName: \"kubernetes.io/projected/61fe3d3b-5d3e-47b5-be4d-46ebd6f02914-kube-api-access-f5qjx\") pod \"maas-api-key-cleanup-29895660-5b2m8\" (UID: \"61fe3d3b-5d3e-47b5-be4d-46ebd6f02914\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-5b2m8"
Nov 03 21:00:00.307702 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.307680    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fe3d3b-5d3e-47b5-be4d-46ebd6f02914-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895660-5b2m8\" (UID: \"61fe3d3b-5d3e-47b5-be4d-46ebd6f02914\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-5b2m8"
Nov 03 21:00:00.320408 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.320376    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5qjx\" (UniqueName: \"kubernetes.io/projected/61fe3d3b-5d3e-47b5-be4d-46ebd6f02914-kube-api-access-f5qjx\") pod \"maas-api-key-cleanup-29895660-5b2m8\" (UID: \"61fe3d3b-5d3e-47b5-be4d-46ebd6f02914\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-5b2m8"
Nov 03 21:00:00.458582 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.458501    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-5b2m8"
Nov 03 21:00:00.733896 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:00.733806    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-5b2m8"]
Nov 03 21:00:00.736742 ip-10-0-143-206 kubenswrapper[5591]: W1103 21:00:00.736706    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61fe3d3b_5d3e_47b5_be4d_46ebd6f02914.slice/crio-ebd6ce934e8b8eb707f9b6e1e5b11684963e0c50925f534920f57e22d56dfa53 WatchSource:0}: Error finding container ebd6ce934e8b8eb707f9b6e1e5b11684963e0c50925f534920f57e22d56dfa53: Status 404 returned error can't find the container with id ebd6ce934e8b8eb707f9b6e1e5b11684963e0c50925f534920f57e22d56dfa53
Nov 03 21:00:01.274999 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:01.274962    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-5b2m8" event={"ID":"61fe3d3b-5d3e-47b5-be4d-46ebd6f02914","Type":"ContainerStarted","Data":"ef0a4469dfcc19f4c87f9054b7985dd99dbd0578aa311ffdc6e51d7a96f46ed8"}
Nov 03 21:00:01.275430 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:01.275021    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-5b2m8" event={"ID":"61fe3d3b-5d3e-47b5-be4d-46ebd6f02914","Type":"ContainerStarted","Data":"ebd6ce934e8b8eb707f9b6e1e5b11684963e0c50925f534920f57e22d56dfa53"}
Nov 03 21:00:01.293717 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:01.293659    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-5b2m8" podStartSLOduration=1.293646543 podStartE2EDuration="1.293646543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:01.293282381 +0000 UTC m=+4950.065575399" watchObservedRunningTime="2026-11-03 21:00:01.293646543 +0000 UTC m=+4950.065939562"
Nov 03 21:00:02.282507 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:02.282465    5591 generic.go:356] "Generic (PLEG): container finished" podID="61fe3d3b-5d3e-47b5-be4d-46ebd6f02914" containerID="ef0a4469dfcc19f4c87f9054b7985dd99dbd0578aa311ffdc6e51d7a96f46ed8" exitCode=60
Nov 03 21:00:02.282956 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:02.282560    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-5b2m8" event={"ID":"61fe3d3b-5d3e-47b5-be4d-46ebd6f02914","Type":"ContainerDied","Data":"ef0a4469dfcc19f4c87f9054b7985dd99dbd0578aa311ffdc6e51d7a96f46ed8"}
Nov 03 21:00:02.282956 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:02.282763    5591 scope.go:122] "RemoveContainer" containerID="ef0a4469dfcc19f4c87f9054b7985dd99dbd0578aa311ffdc6e51d7a96f46ed8"
Nov 03 21:00:03.290029 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:03.289992    5591 generic.go:356] "Generic (PLEG): container finished" podID="61fe3d3b-5d3e-47b5-be4d-46ebd6f02914" containerID="ce384d36ef767cd468ea1617512168cc4a9335bc9a056bd66111c0b21fa98277" exitCode=60
Nov 03 21:00:03.290535 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:03.290085    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-5b2m8" event={"ID":"61fe3d3b-5d3e-47b5-be4d-46ebd6f02914","Type":"ContainerDied","Data":"ce384d36ef767cd468ea1617512168cc4a9335bc9a056bd66111c0b21fa98277"}
Nov 03 21:00:03.290535 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:03.290159    5591 scope.go:122] "RemoveContainer" containerID="ef0a4469dfcc19f4c87f9054b7985dd99dbd0578aa311ffdc6e51d7a96f46ed8"
Nov 03 21:00:03.290535 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:03.290434    5591 scope.go:122] "RemoveContainer" containerID="ce384d36ef767cd468ea1617512168cc4a9335bc9a056bd66111c0b21fa98277"
Nov 03 21:00:03.290705 ip-10-0-143-206 kubenswrapper[5591]: E1103 21:00:03.290687    5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895660-5b2m8_odh-ai-gateway-infra(61fe3d3b-5d3e-47b5-be4d-46ebd6f02914)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-5b2m8" podUID="61fe3d3b-5d3e-47b5-be4d-46ebd6f02914"
Nov 03 21:00:04.297970 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:04.297942    5591 scope.go:122] "RemoveContainer" containerID="ce384d36ef767cd468ea1617512168cc4a9335bc9a056bd66111c0b21fa98277"
Nov 03 21:00:04.298553 ip-10-0-143-206 kubenswrapper[5591]: E1103 21:00:04.298176    5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895660-5b2m8_odh-ai-gateway-infra(61fe3d3b-5d3e-47b5-be4d-46ebd6f02914)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-5b2m8" podUID="61fe3d3b-5d3e-47b5-be4d-46ebd6f02914"
Nov 03 21:00:17.342156 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:17.342071    5591 scope.go:122] "RemoveContainer" containerID="ce384d36ef767cd468ea1617512168cc4a9335bc9a056bd66111c0b21fa98277"
Nov 03 21:00:18.376062 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:18.375971    5591 generic.go:356] "Generic (PLEG): container finished" podID="61fe3d3b-5d3e-47b5-be4d-46ebd6f02914" containerID="f21146bc5301e69085e4ec5ae9bfa04512009f8e8fc8ffbecff37318c2ff9503" exitCode=60
Nov 03 21:00:18.376062 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:18.376045    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-5b2m8" event={"ID":"61fe3d3b-5d3e-47b5-be4d-46ebd6f02914","Type":"ContainerDied","Data":"f21146bc5301e69085e4ec5ae9bfa04512009f8e8fc8ffbecff37318c2ff9503"}
Nov 03 21:00:18.376570 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:18.376109    5591 scope.go:122] "RemoveContainer" containerID="ce384d36ef767cd468ea1617512168cc4a9335bc9a056bd66111c0b21fa98277"
Nov 03 21:00:18.376570 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:18.376465    5591 scope.go:122] "RemoveContainer" containerID="f21146bc5301e69085e4ec5ae9bfa04512009f8e8fc8ffbecff37318c2ff9503"
Nov 03 21:00:18.376742 ip-10-0-143-206 kubenswrapper[5591]: E1103 21:00:18.376721    5591 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895660-5b2m8_odh-ai-gateway-infra(61fe3d3b-5d3e-47b5-be4d-46ebd6f02914)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-5b2m8" podUID="61fe3d3b-5d3e-47b5-be4d-46ebd6f02914"
Nov 03 21:00:19.403259 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:19.403223    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-5b2m8"]
Nov 03 21:00:19.547158 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:19.547132    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-5b2m8"
Nov 03 21:00:19.688887 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:19.688782    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61fe3d3b-5d3e-47b5-be4d-46ebd6f02914-kube-api-access-f5qjx\" (UniqueName: \"kubernetes.io/projected/61fe3d3b-5d3e-47b5-be4d-46ebd6f02914-kube-api-access-f5qjx\") pod \"61fe3d3b-5d3e-47b5-be4d-46ebd6f02914\" (UID: \"61fe3d3b-5d3e-47b5-be4d-46ebd6f02914\") "
Nov 03 21:00:19.689085 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:19.688886    5591 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61fe3d3b-5d3e-47b5-be4d-46ebd6f02914-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fe3d3b-5d3e-47b5-be4d-46ebd6f02914-maas-api-service-ca\") pod \"61fe3d3b-5d3e-47b5-be4d-46ebd6f02914\" (UID: \"61fe3d3b-5d3e-47b5-be4d-46ebd6f02914\") "
Nov 03 21:00:19.689297 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:19.689271    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61fe3d3b-5d3e-47b5-be4d-46ebd6f02914-maas-api-service-ca" pod "61fe3d3b-5d3e-47b5-be4d-46ebd6f02914" (UID: "61fe3d3b-5d3e-47b5-be4d-46ebd6f02914"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 21:00:19.691143 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:19.691114    5591 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61fe3d3b-5d3e-47b5-be4d-46ebd6f02914-kube-api-access-f5qjx" pod "61fe3d3b-5d3e-47b5-be4d-46ebd6f02914" (UID: "61fe3d3b-5d3e-47b5-be4d-46ebd6f02914"). InnerVolumeSpecName "kube-api-access-f5qjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 21:00:19.790496 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:19.790456    5591 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5qjx\" (UniqueName: \"kubernetes.io/projected/61fe3d3b-5d3e-47b5-be4d-46ebd6f02914-kube-api-access-f5qjx\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:19.790496 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:19.790489    5591 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61fe3d3b-5d3e-47b5-be4d-46ebd6f02914-maas-api-service-ca\") on node \"ip-10-0-143-206.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 21:00:20.389884 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:20.389842    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-5b2m8" event={"ID":"61fe3d3b-5d3e-47b5-be4d-46ebd6f02914","Type":"ContainerDied","Data":"ebd6ce934e8b8eb707f9b6e1e5b11684963e0c50925f534920f57e22d56dfa53"}
Nov 03 21:00:20.389884 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:20.389869    5591 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895660-5b2m8"
Nov 03 21:00:20.390133 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:20.389923    5591 scope.go:122] "RemoveContainer" containerID="f21146bc5301e69085e4ec5ae9bfa04512009f8e8fc8ffbecff37318c2ff9503"
Nov 03 21:00:20.412007 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:20.411982    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-5b2m8"]
Nov 03 21:00:20.416856 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:20.416834    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895660-5b2m8"]
Nov 03 21:00:21.348152 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:21.348117    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61fe3d3b-5d3e-47b5-be4d-46ebd6f02914" path="/var/lib/kubelet/pods/61fe3d3b-5d3e-47b5-be4d-46ebd6f02914/volumes"
Nov 03 21:00:28.723452 ip-10-0-143-206 kubenswrapper[5591]: E1103 21:00:28.723406    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 21:00:38.482864 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:38.482828    5591 scope.go:122] "RemoveContainer" containerID="fc44b3eb4743f5461fe91be1a99d160ac1b65bfab18b42c318f1d03f38d28bde"
Nov 03 21:00:38.495130 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:38.495107    5591 scope.go:122] "RemoveContainer" containerID="dd877f497afd282d48d03f3402f6bc5fb73797d62dde81ed29363eaa5b1b2e4a"
Nov 03 21:00:38.504335 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:38.504311    5591 scope.go:122] "RemoveContainer" containerID="573dca7a306adbfe5bb340c5684cedb8ec897c4e331ce69f130da54674cffd61"
Nov 03 21:00:39.433998 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:39.433946    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brh"]
Nov 03 21:00:39.434722 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:39.434691    5591 cpu_manager.go:414] "containerMap: removing container" podUID="61fe3d3b-5d3e-47b5-be4d-46ebd6f02914" containerName="cleanup"
Nov 03 21:00:39.434722 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:39.434711    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61fe3d3b-5d3e-47b5-be4d-46ebd6f02914" containerName="cleanup"
Nov 03 21:00:39.434902 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:39.434743    5591 cpu_manager.go:414] "containerMap: removing container" podUID="61fe3d3b-5d3e-47b5-be4d-46ebd6f02914" containerName="cleanup"
Nov 03 21:00:39.434902 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:39.434749    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61fe3d3b-5d3e-47b5-be4d-46ebd6f02914" containerName="cleanup"
Nov 03 21:00:39.434902 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:39.434812    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="61fe3d3b-5d3e-47b5-be4d-46ebd6f02914" containerName="cleanup"
Nov 03 21:00:39.434902 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:39.434826    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="61fe3d3b-5d3e-47b5-be4d-46ebd6f02914" containerName="cleanup"
Nov 03 21:00:39.434902 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:39.434833    5591 memory_manager.go:365] "RemoveStaleState removing state" podUID="61fe3d3b-5d3e-47b5-be4d-46ebd6f02914" containerName="cleanup"
Nov 03 21:00:39.440032 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:39.440011    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brh"
Nov 03 21:00:39.444081 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:39.444059    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-k488s\""
Nov 03 21:00:39.450712 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:39.450680    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brh"]
Nov 03 21:00:39.484461 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:39.484427    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c83d2c4b-bc16-4d27-8225-cd4d9a587d9a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-b5brh\" (UID: \"c83d2c4b-bc16-4d27-8225-cd4d9a587d9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brh"
Nov 03 21:00:39.484913 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:39.484466    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8nng\" (UniqueName: \"kubernetes.io/projected/c83d2c4b-bc16-4d27-8225-cd4d9a587d9a-kube-api-access-t8nng\") pod \"kuadrant-operator-controller-manager-678d87766f-b5brh\" (UID: \"c83d2c4b-bc16-4d27-8225-cd4d9a587d9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brh"
Nov 03 21:00:39.585089 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:39.585047    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c83d2c4b-bc16-4d27-8225-cd4d9a587d9a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-b5brh\" (UID: \"c83d2c4b-bc16-4d27-8225-cd4d9a587d9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brh"
Nov 03 21:00:39.585089 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:39.585091    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8nng\" (UniqueName: \"kubernetes.io/projected/c83d2c4b-bc16-4d27-8225-cd4d9a587d9a-kube-api-access-t8nng\") pod \"kuadrant-operator-controller-manager-678d87766f-b5brh\" (UID: \"c83d2c4b-bc16-4d27-8225-cd4d9a587d9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brh"
Nov 03 21:00:39.585532 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:39.585506    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c83d2c4b-bc16-4d27-8225-cd4d9a587d9a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-b5brh\" (UID: \"c83d2c4b-bc16-4d27-8225-cd4d9a587d9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brh"
Nov 03 21:00:39.598045 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:39.598008    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8nng\" (UniqueName: \"kubernetes.io/projected/c83d2c4b-bc16-4d27-8225-cd4d9a587d9a-kube-api-access-t8nng\") pod \"kuadrant-operator-controller-manager-678d87766f-b5brh\" (UID: \"c83d2c4b-bc16-4d27-8225-cd4d9a587d9a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brh"
Nov 03 21:00:39.754367 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:39.754265    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brh"
Nov 03 21:00:40.038953 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:40.038924    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brh"]
Nov 03 21:00:40.042303 ip-10-0-143-206 kubenswrapper[5591]: W1103 21:00:40.042271    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc83d2c4b_bc16_4d27_8225_cd4d9a587d9a.slice/crio-3ddb651f340c7b0acf45c4b06c8bdbd5475390511f7be1fcd10e604567009e54 WatchSource:0}: Error finding container 3ddb651f340c7b0acf45c4b06c8bdbd5475390511f7be1fcd10e604567009e54: Status 404 returned error can't find the container with id 3ddb651f340c7b0acf45c4b06c8bdbd5475390511f7be1fcd10e604567009e54
Nov 03 21:00:40.044784 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:40.044757    5591 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 21:00:40.501532 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:40.501444    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brh" event={"ID":"c83d2c4b-bc16-4d27-8225-cd4d9a587d9a","Type":"ContainerStarted","Data":"3ddb651f340c7b0acf45c4b06c8bdbd5475390511f7be1fcd10e604567009e54"}
Nov 03 21:00:46.538576 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:46.538535    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brh" event={"ID":"c83d2c4b-bc16-4d27-8225-cd4d9a587d9a","Type":"ContainerStarted","Data":"e5083f63b5959684321ec9981a13912692ee0426643bdf7615d15947042ad17d"}
Nov 03 21:00:46.539013 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:46.538627    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brh"
Nov 03 21:00:46.561846 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:46.561782    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brh" podStartSLOduration=1.6952878789999999 podStartE2EDuration="7.561769416s" totalImagesPullingTime="5.866481537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:46.560669289 +0000 UTC m=+4995.332962308" watchObservedRunningTime="2026-11-03 21:00:46.561769416 +0000 UTC m=+4995.334062434"
Nov 03 21:00:57.546832 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:00:57.546800    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-b5brh"
Nov 03 21:00:58.722935 ip-10-0-143-206 kubenswrapper[5591]: E1103 21:00:58.722895    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 21:01:03.941767 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:03.941730    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 21:01:18.143946 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:18.143907    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 21:01:21.869128 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:21.869065    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 21:01:25.185619 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:25.185581    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 21:01:28.723508 ip-10-0-143-206 kubenswrapper[5591]: E1103 21:01:28.723465    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 21:01:29.022940 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:29.022904    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-62bn4"]
Nov 03 21:01:29.025504 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:29.025473    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-62bn4"]
Nov 03 21:01:29.348487 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:29.348382    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77" path="/var/lib/kubelet/pods/1a5fe312-e47a-42e9-b8f9-32ea8f9c4a77/volumes"
Nov 03 21:01:38.538840 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:38.538803    5591 scope.go:122] "RemoveContainer" containerID="764ca844cd5a949320145b8303feae4dc71e61d83a26d8ff9c9df1881e16857d"
Nov 03 21:01:38.550505 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:38.550480    5591 scope.go:122] "RemoveContainer" containerID="f17f46a45968ef6598bafac11714400bb5ccf428af957141cd7778c571ff6e1f"
Nov 03 21:01:38.560065 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:38.560032    5591 scope.go:122] "RemoveContainer" containerID="de5b14d436bb1765a99c39b6cc7580a416768269d1b342cbfc3a5176eb3f4415"
Nov 03 21:01:43.331454 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:43.331361    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c4b55fdfd-kqszv"]
Nov 03 21:01:43.332064 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:43.332034    5591 cpu_manager.go:414] "containerMap: removing container" podUID="61fe3d3b-5d3e-47b5-be4d-46ebd6f02914" containerName="cleanup"
Nov 03 21:01:43.332064 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:43.332063    5591 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61fe3d3b-5d3e-47b5-be4d-46ebd6f02914" containerName="cleanup"
Nov 03 21:01:43.335466 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:43.335448    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-kqszv"
Nov 03 21:01:43.339778 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:43.339754    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 21:01:43.341036 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:43.341010    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-942q2\""
Nov 03 21:01:43.341140 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:43.341028    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 21:01:43.341140 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:43.341099    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 21:01:43.341485 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:43.341467    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 21:01:43.348924 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:43.348902    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c4b55fdfd-kqszv"]
Nov 03 21:01:43.505693 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:43.505650    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66ef7a59-6b17-45b4-b050-f575d62e903e-metrics-certs\") pod \"maas-api-5c4b55fdfd-kqszv\" (UID: \"66ef7a59-6b17-45b4-b050-f575d62e903e\") " pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-kqszv"
Nov 03 21:01:43.505874 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:43.505715    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjzjv\" (UniqueName: \"kubernetes.io/projected/66ef7a59-6b17-45b4-b050-f575d62e903e-kube-api-access-pjzjv\") pod \"maas-api-5c4b55fdfd-kqszv\" (UID: \"66ef7a59-6b17-45b4-b050-f575d62e903e\") " pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-kqszv"
Nov 03 21:01:43.505919 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:43.505869    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ef7a59-6b17-45b4-b050-f575d62e903e-openshift-service-ca\") pod \"maas-api-5c4b55fdfd-kqszv\" (UID: \"66ef7a59-6b17-45b4-b050-f575d62e903e\") " pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-kqszv"
Nov 03 21:01:43.505964 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:43.505917    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66ef7a59-6b17-45b4-b050-f575d62e903e-maas-api-tls\") pod \"maas-api-5c4b55fdfd-kqszv\" (UID: \"66ef7a59-6b17-45b4-b050-f575d62e903e\") " pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-kqszv"
Nov 03 21:01:43.607014 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:43.606921    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ef7a59-6b17-45b4-b050-f575d62e903e-openshift-service-ca\") pod \"maas-api-5c4b55fdfd-kqszv\" (UID: \"66ef7a59-6b17-45b4-b050-f575d62e903e\") " pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-kqszv"
Nov 03 21:01:43.607014 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:43.606974    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66ef7a59-6b17-45b4-b050-f575d62e903e-maas-api-tls\") pod \"maas-api-5c4b55fdfd-kqszv\" (UID: \"66ef7a59-6b17-45b4-b050-f575d62e903e\") " pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-kqszv"
Nov 03 21:01:43.607266 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:43.607083    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66ef7a59-6b17-45b4-b050-f575d62e903e-metrics-certs\") pod \"maas-api-5c4b55fdfd-kqszv\" (UID: \"66ef7a59-6b17-45b4-b050-f575d62e903e\") " pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-kqszv"
Nov 03 21:01:43.607266 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:43.607142    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjzjv\" (UniqueName: \"kubernetes.io/projected/66ef7a59-6b17-45b4-b050-f575d62e903e-kube-api-access-pjzjv\") pod \"maas-api-5c4b55fdfd-kqszv\" (UID: \"66ef7a59-6b17-45b4-b050-f575d62e903e\") " pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-kqszv"
Nov 03 21:01:43.607883 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:43.607786    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ef7a59-6b17-45b4-b050-f575d62e903e-openshift-service-ca\") pod \"maas-api-5c4b55fdfd-kqszv\" (UID: \"66ef7a59-6b17-45b4-b050-f575d62e903e\") " pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-kqszv"
Nov 03 21:01:43.609715 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:43.609688    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66ef7a59-6b17-45b4-b050-f575d62e903e-maas-api-tls\") pod \"maas-api-5c4b55fdfd-kqszv\" (UID: \"66ef7a59-6b17-45b4-b050-f575d62e903e\") " pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-kqszv"
Nov 03 21:01:43.609936 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:43.609914    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66ef7a59-6b17-45b4-b050-f575d62e903e-metrics-certs\") pod \"maas-api-5c4b55fdfd-kqszv\" (UID: \"66ef7a59-6b17-45b4-b050-f575d62e903e\") " pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-kqszv"
Nov 03 21:01:43.620577 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:43.620546    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjzjv\" (UniqueName: \"kubernetes.io/projected/66ef7a59-6b17-45b4-b050-f575d62e903e-kube-api-access-pjzjv\") pod \"maas-api-5c4b55fdfd-kqszv\" (UID: \"66ef7a59-6b17-45b4-b050-f575d62e903e\") " pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-kqszv"
Nov 03 21:01:43.652296 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:43.652255    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-kqszv"
Nov 03 21:01:43.932825 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:43.932671    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c4b55fdfd-kqszv"]
Nov 03 21:01:43.935261 ip-10-0-143-206 kubenswrapper[5591]: W1103 21:01:43.935236    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ef7a59_6b17_45b4_b050_f575d62e903e.slice/crio-3b6796c15edc099760ec42c6c0fad1bd0483e839baca7b7e23c9b3ee448d5a0b WatchSource:0}: Error finding container 3b6796c15edc099760ec42c6c0fad1bd0483e839baca7b7e23c9b3ee448d5a0b: Status 404 returned error can't find the container with id 3b6796c15edc099760ec42c6c0fad1bd0483e839baca7b7e23c9b3ee448d5a0b
Nov 03 21:01:44.891180 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:44.891141    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-kqszv" event={"ID":"66ef7a59-6b17-45b4-b050-f575d62e903e","Type":"ContainerStarted","Data":"41ba731494c5edda1ca098e0cf046db8ac116646b145a4113e7e2bde5921f9dd"}
Nov 03 21:01:44.891654 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:44.891226    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-kqszv" event={"ID":"66ef7a59-6b17-45b4-b050-f575d62e903e","Type":"ContainerStarted","Data":"3b6796c15edc099760ec42c6c0fad1bd0483e839baca7b7e23c9b3ee448d5a0b"}
Nov 03 21:01:44.891654 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:44.891361    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-kqszv"
Nov 03 21:01:44.911566 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:44.911512    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-kqszv" podStartSLOduration=1.51127481 podStartE2EDuration="1.91149564s" totalImagesPullingTime="400.22083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:44.911005055 +0000 UTC m=+5053.683298072" watchObservedRunningTime="2026-11-03 21:01:44.91149564 +0000 UTC m=+5053.683788658"
Nov 03 21:01:49.021007 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:49.020968    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j6znv"]
Nov 03 21:01:49.023511 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:49.023478    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-j6znv"]
Nov 03 21:01:49.348120 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:49.348040    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48d9ca25-d5a6-4fce-b0f8-11b395a188f5" path="/var/lib/kubelet/pods/48d9ca25-d5a6-4fce-b0f8-11b395a188f5/volumes"
Nov 03 21:01:50.901902 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:50.901872    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c4b55fdfd-kqszv"
Nov 03 21:01:54.092016 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:54.091980    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 21:01:56.784229 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:56.780356    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-mrwmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:01:56.784229 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:01:56.780420    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt" podUID="e14ca8f3-c80b-4c0b-9564-ef9734b86de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:01:58.739429 ip-10-0-143-206 kubenswrapper[5591]: E1103 21:01:58.739385    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 21:02:28.723844 ip-10-0-143-206 kubenswrapper[5591]: E1103 21:02:28.723803    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 21:02:34.302938 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:02:34.302901    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 21:02:34.336286 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:02:34.336257    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 21:02:34.556874 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:02:34.556792    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 21:02:37.649333 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:02:37.649299    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 21:02:38.592348 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:02:38.592307    5591 scope.go:122] "RemoveContainer" containerID="2ae0a58b6850ddff351468414babed9d4fefc83b3575ca520695d393f79ef97e"
Nov 03 21:02:38.612325 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:02:38.612291    5591 scope.go:122] "RemoveContainer" containerID="a284057cf51f343dbb001f04102d88d2fa2710882f090e55a3628b8393e71665"
Nov 03 21:02:54.367232 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:02:54.367171    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 21:02:58.855807 ip-10-0-143-206 kubenswrapper[5591]: E1103 21:02:58.855734    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 21:02:59.530423 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:02:59.530383    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 21:03:15.022371 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:03:15.022289    5591 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1fba839-cxn8c"]
Nov 03 21:03:15.028320 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:03:15.028288    5591 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1fba839-cxn8c"]
Nov 03 21:03:15.347603 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:03:15.347517    5591 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6" path="/var/lib/kubelet/pods/7b7d4627-fe8b-4bd4-a6a8-49c3f9b149c6/volumes"
Nov 03 21:03:20.558418 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:03:20.558385    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 21:03:28.726402 ip-10-0-143-206 kubenswrapper[5591]: E1103 21:03:28.726349    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 21:03:30.751512 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:03:30.751459    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 21:03:35.807046 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:03:35.806980    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 21:03:38.650054 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:03:38.650018    5591 scope.go:122] "RemoveContainer" containerID="06b9d0e200b23e89e08757d7e56fe8ca662934aa772bf28e600b7768a3ed54fe"
Nov 03 21:03:38.672823 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:03:38.672799    5591 scope.go:122] "RemoveContainer" containerID="f32909444d05585d8ee8b2a65bbb165312965d9c0655d0cbdf53435f23a3cba9"
Nov 03 21:03:41.078611 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:03:41.078568    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 21:03:57.882983 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:03:57.882944    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 21:03:58.739170 ip-10-0-143-206 kubenswrapper[5591]: E1103 21:03:58.739123    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 21:04:08.148171 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:04:08.148128    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 21:04:11.727230 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:04:11.727108    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 21:04:22.966768 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:04:22.966719    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 21:04:28.723582 ip-10-0-143-206 kubenswrapper[5591]: E1103 21:04:28.723537    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 21:04:38.739165 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:04:38.739131    5591 scope.go:122] "RemoveContainer" containerID="d20bf423f5ec130bff23a1e070396922c71970d35a4405672956312523dea68a"
Nov 03 21:04:38.862728 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:04:38.862691    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 21:04:41.782422 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:04:41.782314    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-mrwmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:04:41.783077 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:04:41.783042    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt" podUID="e14ca8f3-c80b-4c0b-9564-ef9734b86de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:04:58.724881 ip-10-0-143-206 kubenswrapper[5591]: E1103 21:04:58.724836    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 21:05:11.790232 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:05:11.787327    5591 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-mrwmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 21:05:11.790232 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:05:11.787396    5591 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-mrwmt" podUID="e14ca8f3-c80b-4c0b-9564-ef9734b86de0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 21:05:28.725013 ip-10-0-143-206 kubenswrapper[5591]: E1103 21:05:28.724972    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 21:05:38.759713 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:05:38.759676    5591 scope.go:122] "RemoveContainer" containerID="ce5754ac4f9fbfb0c90571d6c04fc2082657e608e09e715cbb921e1dc040f364"
Nov 03 21:05:38.769316 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:05:38.769291    5591 scope.go:122] "RemoveContainer" containerID="e43a449c61345d7452b74822b2c8e4bc1c70e8807ffd865a8e2c52f474af7140"
Nov 03 21:05:58.724220 ip-10-0-143-206 kubenswrapper[5591]: E1103 21:05:58.724163    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 21:06:28.723265 ip-10-0-143-206 kubenswrapper[5591]: E1103 21:06:28.723198    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 21:06:30.662930 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:06:30.662882    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-fz8hm"]
Nov 03 21:06:58.724419 ip-10-0-143-206 kubenswrapper[5591]: E1103 21:06:58.724374    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 21:07:28.724028 ip-10-0-143-206 kubenswrapper[5591]: E1103 21:07:28.723985    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 21:07:34.336958 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:07:34.336833    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 21:07:34.369515 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:07:34.369483    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x88xn_65b590ed-e1a5-467f-9283-7c875dd9ee24/ovn-acl-logging/0.log"
Nov 03 21:07:41.118746 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:07:41.118702    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-79b8b796f-2sshp_747740b4-92e0-4179-9318-04fd9030b8e0/manager/0.log"
Nov 03 21:07:41.346467 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:07:41.346365    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-vx6gv_67d4147a-49d0-43ad-983d-cc033e4fc21a/manager/0.log"
Nov 03 21:07:43.603647 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:07:43.603618    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5c4b55fdfd-kqszv_66ef7a59-6b17-45b4-b050-f575d62e903e/maas-api/0.log"
Nov 03 21:07:49.740736 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:07:49.740696    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-b5brh_c83d2c4b-bc16-4d27-8225-cd4d9a587d9a/manager/0.log"
Nov 03 21:07:49.981500 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:07:49.981466    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-fz8hm_ea095e27-20df-40bb-a4c4-3997bd5e9cff/limitador/0.log"
Nov 03 21:07:51.551013 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:07:51.550980    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79ddfb4884-rnf5q_b4cff30c-a64b-41d5-a6e8-62241d1c7a67/kube-auth-proxy/0.log"
Nov 03 21:07:52.697835 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:07:52.697791    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-mrwmt_e14ca8f3-c80b-4c0b-9564-ef9734b86de0/istio-proxy/0.log"
Nov 03 21:07:53.485877 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:07:53.485822    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-x4bds_128a2ead-106d-483e-a4e0-8e13c7c91e28/istio-proxy/0.log"
Nov 03 21:07:53.682631 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:07:53.682594    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-zst5d_e8c03ef4-fb42-4a08-b02d-98165eb9b203/istio-proxy/0.log"
Nov 03 21:07:54.249364 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:07:54.249320    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-68d58ff5c7-z4hlv_990d3b87-4fac-45c8-862b-f8dca1606467/payload-processing/0.log"
Nov 03 21:07:55.569949 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:07:55.569892    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-4p6wg_dc960ce4-0839-4aa4-b30c-c64f550b1655/main/0.log"
Nov 03 21:07:55.938072 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:07:55.937985    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-c2fbm_0de7df58-cc36-454f-8954-fb6cb562da6a/main/0.log"
Nov 03 21:07:56.121699 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:07:56.121664    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-fxm7k_6efbc03a-f9b7-463d-969e-7c420f0d59e1/main/0.log"
Nov 03 21:07:56.676151 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:07:56.676114    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-krbks_8c1de34b-e0ff-4411-b4c1-1318253b182a/main/0.log"
Nov 03 21:07:58.723978 ip-10-0-143-206 kubenswrapper[5591]: E1103 21:07:58.723938    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 21:08:06.865272 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:06.865234    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-phpvf_956c41e5-c95c-4dc4-8734-99bd8cecd1be/main/0.log"
Nov 03 21:08:20.679830 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:20.679789    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6q5vq_98d32fbe-5011-47a8-b937-725a8cad95a4/global-pull-secret-syncer/0.log"
Nov 03 21:08:20.847714 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:20.847687    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-9pxsv_4b60e6cd-0733-43f9-ac06-b7183f8ddbe6/connection-checker/0.log"
Nov 03 21:08:20.875558 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:20.875530    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-gwxd6_1fe14dc7-e65e-4115-a93a-04719aa38622/connection-checker/0.log"
Nov 03 21:08:20.902747 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:20.902722    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-h8wps_585afe81-6315-4c26-b8fb-2caa9c6c121a/connection-checker/0.log"
Nov 03 21:08:21.112748 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:21.112706    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vh26d_b1cda986-f386-462d-9956-acc2a4de6e32/konnectivity-agent/0.log"
Nov 03 21:08:21.222410 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:21.222374    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-206.us-east-2.compute.internal_3def2722c7c58bcddedb41ddbc48c3ab/haproxy/0.log"
Nov 03 21:08:27.795415 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:27.795371    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-b5brh_c83d2c4b-bc16-4d27-8225-cd4d9a587d9a/manager/0.log"
Nov 03 21:08:27.824497 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:27.824458    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-fz8hm_ea095e27-20df-40bb-a4c4-3997bd5e9cff/limitador/0.log"
Nov 03 21:08:28.724821 ip-10-0-143-206 kubenswrapper[5591]: E1103 21:08:28.724778    5591 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2224c" containerName="extractor"
Nov 03 21:08:29.360901 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:29.360869    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2dda2e6-5ea0-4469-9cf1-403560cfc3cb/alertmanager/0.log"
Nov 03 21:08:29.391133 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:29.391105    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2dda2e6-5ea0-4469-9cf1-403560cfc3cb/config-reloader/0.log"
Nov 03 21:08:29.416316 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:29.416281    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2dda2e6-5ea0-4469-9cf1-403560cfc3cb/kube-rbac-proxy-web/0.log"
Nov 03 21:08:29.441830 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:29.441791    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2dda2e6-5ea0-4469-9cf1-403560cfc3cb/kube-rbac-proxy/0.log"
Nov 03 21:08:29.467362 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:29.467332    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2dda2e6-5ea0-4469-9cf1-403560cfc3cb/kube-rbac-proxy-metric/0.log"
Nov 03 21:08:29.493108 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:29.493076    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2dda2e6-5ea0-4469-9cf1-403560cfc3cb/prom-label-proxy/0.log"
Nov 03 21:08:29.518685 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:29.518655    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a2dda2e6-5ea0-4469-9cf1-403560cfc3cb/init-config-reloader/0.log"
Nov 03 21:08:29.590250 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:29.590169    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-t8nqk_300a6502-6932-49fe-936a-ba0c16978c94/kube-state-metrics/0.log"
Nov 03 21:08:29.610047 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:29.610011    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-t8nqk_300a6502-6932-49fe-936a-ba0c16978c94/kube-rbac-proxy-main/0.log"
Nov 03 21:08:29.646491 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:29.646400    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-t8nqk_300a6502-6932-49fe-936a-ba0c16978c94/kube-rbac-proxy-self/0.log"
Nov 03 21:08:29.708803 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:29.708772    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-bnrmk_f6bc2cf5-3e45-4217-af83-38df92cfee7f/monitoring-plugin/0.log"
Nov 03 21:08:29.944162 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:29.944078    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sckcx_803910f1-a1f3-4316-ada9-1890e46bbda1/node-exporter/0.log"
Nov 03 21:08:29.968996 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:29.968966    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sckcx_803910f1-a1f3-4316-ada9-1890e46bbda1/kube-rbac-proxy/0.log"
Nov 03 21:08:29.993719 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:29.993692    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sckcx_803910f1-a1f3-4316-ada9-1890e46bbda1/init-textfile/0.log"
Nov 03 21:08:30.312346 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.312310    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rvqts_1cd0db81-8027-4c6a-b33c-3b4545351eac/prometheus-operator/0.log"
Nov 03 21:08:30.337698 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.337672    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rvqts_1cd0db81-8027-4c6a-b33c-3b4545351eac/kube-rbac-proxy/0.log"
Nov 03 21:08:30.365511 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.365470    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-6x2hv_dc03a37b-a84b-4df5-9306-57b8e1ec8f7b/prometheus-operator-admission-webhook/0.log"
Nov 03 21:08:30.495826 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.495781    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bfbb78f4c-2gsct_c7e50fe5-0dd0-4f59-bc5f-99199500790b/thanos-query/0.log"
Nov 03 21:08:30.526518 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.526491    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bfbb78f4c-2gsct_c7e50fe5-0dd0-4f59-bc5f-99199500790b/kube-rbac-proxy-web/0.log"
Nov 03 21:08:30.552880 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.552849    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bfbb78f4c-2gsct_c7e50fe5-0dd0-4f59-bc5f-99199500790b/kube-rbac-proxy/0.log"
Nov 03 21:08:30.579559 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.579474    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bfbb78f4c-2gsct_c7e50fe5-0dd0-4f59-bc5f-99199500790b/prom-label-proxy/0.log"
Nov 03 21:08:30.604452 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.604421    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bfbb78f4c-2gsct_c7e50fe5-0dd0-4f59-bc5f-99199500790b/kube-rbac-proxy-rules/0.log"
Nov 03 21:08:30.620845 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.620812    5591 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c46p7/perf-node-gather-daemonset-clbc7"]
Nov 03 21:08:30.629934 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.629899    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-clbc7"
Nov 03 21:08:30.634904 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.634874    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c46p7/perf-node-gather-daemonset-clbc7"]
Nov 03 21:08:30.639175 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.639148    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c46p7\"/\"openshift-service-ca.crt\""
Nov 03 21:08:30.640412 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.640387    5591 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c46p7\"/\"kube-root-ca.crt\""
Nov 03 21:08:30.640596 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.640474    5591 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c46p7\"/\"default-dockercfg-dfcj5\""
Nov 03 21:08:30.642394 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.642371    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bfbb78f4c-2gsct_c7e50fe5-0dd0-4f59-bc5f-99199500790b/kube-rbac-proxy-metrics/0.log"
Nov 03 21:08:30.669161 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.669129    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2a0fff9-e1f4-4d19-86b0-320171740d83-sys\") pod \"perf-node-gather-daemonset-clbc7\" (UID: \"a2a0fff9-e1f4-4d19-86b0-320171740d83\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-clbc7"
Nov 03 21:08:30.669371 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.669175    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2a0fff9-e1f4-4d19-86b0-320171740d83-podres\") pod \"perf-node-gather-daemonset-clbc7\" (UID: \"a2a0fff9-e1f4-4d19-86b0-320171740d83\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-clbc7"
Nov 03 21:08:30.669371 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.669198    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2a0fff9-e1f4-4d19-86b0-320171740d83-lib-modules\") pod \"perf-node-gather-daemonset-clbc7\" (UID: \"a2a0fff9-e1f4-4d19-86b0-320171740d83\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-clbc7"
Nov 03 21:08:30.669371 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.669339    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq5bj\" (UniqueName: \"kubernetes.io/projected/a2a0fff9-e1f4-4d19-86b0-320171740d83-kube-api-access-nq5bj\") pod \"perf-node-gather-daemonset-clbc7\" (UID: \"a2a0fff9-e1f4-4d19-86b0-320171740d83\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-clbc7"
Nov 03 21:08:30.669553 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.669506    5591 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2a0fff9-e1f4-4d19-86b0-320171740d83-proc\") pod \"perf-node-gather-daemonset-clbc7\" (UID: \"a2a0fff9-e1f4-4d19-86b0-320171740d83\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-clbc7"
Nov 03 21:08:30.770714 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.770668    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq5bj\" (UniqueName: \"kubernetes.io/projected/a2a0fff9-e1f4-4d19-86b0-320171740d83-kube-api-access-nq5bj\") pod \"perf-node-gather-daemonset-clbc7\" (UID: \"a2a0fff9-e1f4-4d19-86b0-320171740d83\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-clbc7"
Nov 03 21:08:30.770946 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.770778    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2a0fff9-e1f4-4d19-86b0-320171740d83-proc\") pod \"perf-node-gather-daemonset-clbc7\" (UID: \"a2a0fff9-e1f4-4d19-86b0-320171740d83\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-clbc7"
Nov 03 21:08:30.770946 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.770800    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2a0fff9-e1f4-4d19-86b0-320171740d83-sys\") pod \"perf-node-gather-daemonset-clbc7\" (UID: \"a2a0fff9-e1f4-4d19-86b0-320171740d83\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-clbc7"
Nov 03 21:08:30.770946 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.770830    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2a0fff9-e1f4-4d19-86b0-320171740d83-podres\") pod \"perf-node-gather-daemonset-clbc7\" (UID: \"a2a0fff9-e1f4-4d19-86b0-320171740d83\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-clbc7"
Nov 03 21:08:30.770946 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.770851    5591 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2a0fff9-e1f4-4d19-86b0-320171740d83-lib-modules\") pod \"perf-node-gather-daemonset-clbc7\" (UID: \"a2a0fff9-e1f4-4d19-86b0-320171740d83\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-clbc7"
Nov 03 21:08:30.770946 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.770910    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2a0fff9-e1f4-4d19-86b0-320171740d83-proc\") pod \"perf-node-gather-daemonset-clbc7\" (UID: \"a2a0fff9-e1f4-4d19-86b0-320171740d83\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-clbc7"
Nov 03 21:08:30.770946 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.770926    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2a0fff9-e1f4-4d19-86b0-320171740d83-sys\") pod \"perf-node-gather-daemonset-clbc7\" (UID: \"a2a0fff9-e1f4-4d19-86b0-320171740d83\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-clbc7"
Nov 03 21:08:30.770946 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.770948    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2a0fff9-e1f4-4d19-86b0-320171740d83-lib-modules\") pod \"perf-node-gather-daemonset-clbc7\" (UID: \"a2a0fff9-e1f4-4d19-86b0-320171740d83\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-clbc7"
Nov 03 21:08:30.771262 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.770948    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2a0fff9-e1f4-4d19-86b0-320171740d83-podres\") pod \"perf-node-gather-daemonset-clbc7\" (UID: \"a2a0fff9-e1f4-4d19-86b0-320171740d83\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-clbc7"
Nov 03 21:08:30.783707 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.783674    5591 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq5bj\" (UniqueName: \"kubernetes.io/projected/a2a0fff9-e1f4-4d19-86b0-320171740d83-kube-api-access-nq5bj\") pod \"perf-node-gather-daemonset-clbc7\" (UID: \"a2a0fff9-e1f4-4d19-86b0-320171740d83\") " pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-clbc7"
Nov 03 21:08:30.945279 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:30.945153    5591 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-clbc7"
Nov 03 21:08:31.223895 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:31.223720    5591 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c46p7/perf-node-gather-daemonset-clbc7"]
Nov 03 21:08:31.227572 ip-10-0-143-206 kubenswrapper[5591]: W1103 21:08:31.227544    5591 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda2a0fff9_e1f4_4d19_86b0_320171740d83.slice/crio-f5adc0d9257be6dc8e3c98a854b699bc515a06f753612e9e84838b957eeab25d WatchSource:0}: Error finding container f5adc0d9257be6dc8e3c98a854b699bc515a06f753612e9e84838b957eeab25d: Status 404 returned error can't find the container with id f5adc0d9257be6dc8e3c98a854b699bc515a06f753612e9e84838b957eeab25d
Nov 03 21:08:31.315294 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:31.315256    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-clbc7" event={"ID":"a2a0fff9-e1f4-4d19-86b0-320171740d83","Type":"ContainerStarted","Data":"fde425b99fa34f57642e638148c13306bf2827c80517b2be74386fd50c6811c9"}
Nov 03 21:08:31.315481 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:31.315316    5591 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-clbc7" event={"ID":"a2a0fff9-e1f4-4d19-86b0-320171740d83","Type":"ContainerStarted","Data":"f5adc0d9257be6dc8e3c98a854b699bc515a06f753612e9e84838b957eeab25d"}
Nov 03 21:08:31.315481 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:31.315342    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-clbc7"
Nov 03 21:08:31.335713 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:31.335654    5591 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-clbc7" podStartSLOduration=1.335637118 podStartE2EDuration="1.335637118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:08:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:31.334323152 +0000 UTC m=+5460.106616171" watchObservedRunningTime="2026-11-03 21:08:31.335637118 +0000 UTC m=+5460.107930135"
Nov 03 21:08:36.423296 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:36.423262    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-868lx_3ac8ac4b-fef2-475d-a40b-9db77cded4d3/dns/0.log"
Nov 03 21:08:36.446197 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:36.446165    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-868lx_3ac8ac4b-fef2-475d-a40b-9db77cded4d3/kube-rbac-proxy/0.log"
Nov 03 21:08:36.582843 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:36.582807    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rzn4t_630922bd-4efc-4b44-96d1-fa55bce60e21/dns-node-resolver/0.log"
Nov 03 21:08:37.319063 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:37.319010    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6c57db574-5845n_dac5ea76-74c5-41d8-82bc-8285bc87e535/registry/0.log"
Nov 03 21:08:37.331602 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:37.331574    5591 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-c46p7/perf-node-gather-daemonset-clbc7"
Nov 03 21:08:37.412490 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:37.412455    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pswxd_88a27c5e-de31-4090-be4d-6b8d6551e4c7/node-ca/0.log"
Nov 03 21:08:39.588631 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:39.588600    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-79b8b796f-2sshp_747740b4-92e0-4179-9318-04fd9030b8e0/manager/0.log"
Nov 03 21:08:39.652116 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:39.652084    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-vx6gv_67d4147a-49d0-43ad-983d-cc033e4fc21a/manager/0.log"
Nov 03 21:08:40.928182 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:40.928151    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79ddfb4884-rnf5q_b4cff30c-a64b-41d5-a6e8-62241d1c7a67/kube-auth-proxy/0.log"
Nov 03 21:08:41.184922 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:41.184797    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-mrwmt_e14ca8f3-c80b-4c0b-9564-ef9734b86de0/istio-proxy/0.log"
Nov 03 21:08:41.337883 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:41.337856    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-x4bds_128a2ead-106d-483e-a4e0-8e13c7c91e28/istio-proxy/0.log"
Nov 03 21:08:41.382456 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:41.382422    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-zst5d_e8c03ef4-fb42-4a08-b02d-98165eb9b203/istio-proxy/0.log"
Nov 03 21:08:41.506461 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:41.506374    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-68d58ff5c7-z4hlv_990d3b87-4fac-45c8-862b-f8dca1606467/payload-processing/0.log"
Nov 03 21:08:42.768505 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:42.768458    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-79b8b796f-2sshp_747740b4-92e0-4179-9318-04fd9030b8e0/manager/0.log"
Nov 03 21:08:42.925881 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:42.925849    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-vx6gv_67d4147a-49d0-43ad-983d-cc033e4fc21a/manager/0.log"
Nov 03 21:08:45.140675 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:45.140642    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mksqw_f926eeff-6488-48c5-9e97-269a6d79dee1/serve-healthcheck-canary/0.log"
Nov 03 21:08:45.168317 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:45.168281    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79ddfb4884-rnf5q_b4cff30c-a64b-41d5-a6e8-62241d1c7a67/kube-auth-proxy/0.log"
Nov 03 21:08:45.314023 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:45.313989    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-mrwmt_e14ca8f3-c80b-4c0b-9564-ef9734b86de0/istio-proxy/0.log"
Nov 03 21:08:45.397608 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:45.397493    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-x4bds_128a2ead-106d-483e-a4e0-8e13c7c91e28/istio-proxy/0.log"
Nov 03 21:08:45.420077 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:45.420046    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-zst5d_e8c03ef4-fb42-4a08-b02d-98165eb9b203/istio-proxy/0.log"
Nov 03 21:08:45.469453 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:45.469423    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-68d58ff5c7-z4hlv_990d3b87-4fac-45c8-862b-f8dca1606467/payload-processing/0.log"
Nov 03 21:08:45.660056 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:45.659953    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-79b8b796f-2sshp_747740b4-92e0-4179-9318-04fd9030b8e0/manager/0.log"
Nov 03 21:08:45.672171 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:45.672137    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-vx6gv_67d4147a-49d0-43ad-983d-cc033e4fc21a/manager/0.log"
Nov 03 21:08:45.803124 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:45.803092    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2224c_8771d72e-12a3-429f-b458-e22298238393/kube-rbac-proxy/0.log"
Nov 03 21:08:45.830721 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:45.830688    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2224c_8771d72e-12a3-429f-b458-e22298238393/exporter/0.log"
Nov 03 21:08:45.858857 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:45.858821    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2224c_8771d72e-12a3-429f-b458-e22298238393/extractor/0.log"
Nov 03 21:08:48.854027 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:48.853988    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-8464c976fd-8g6hn_e0374e84-add2-4cdc-885a-4e1c130cedd7/manager/0.log"
Nov 03 21:08:48.877992 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:48.877943    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-8464c976fd-8g6hn_e0374e84-add2-4cdc-885a-4e1c130cedd7/copy-manifests/0.log"
Nov 03 21:08:49.985092 ip-10-0-143-206 kubenswrapper[5591]: I1103 21:08:49.985052    5591 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79ddfb4884-rnf5q_b4cff30c-a64b-41d5-a6e8-62241d1c7a67/kube-auth-proxy/0.log"